AppReferences for Appendices

# Money, Time, and Grant Design

Kyle Myers Wei Yang Tham Harvard Business School & Laboratory for Innovation
Science at Harvard<EMAIL_ADDRESS>Business School & Laboratory for
Innovation Science at Harvard<EMAIL_ADDRESS>
This project received financial support from The Alfred P. Sloan Foundation
and the Harvard Business School. This project would not have been possible
without the excellent work of Rachel Mural, Nina Cohodes, and Yilun Xu as well
as input from Marie Thursby, Jerry Thursby, and Karim Lakhani. We have
received helpful comments from Amitabh Chandra, Matt Clancy, Ryan Hill,
Jeffrey Furman, Fabian Gaessler, Joshua Graff Zivin, Benjamin Jones, Megan
MacGarvie, Gustavo Manso, Ramana Nanda, Sam Ransbotham, Henry Sauermann, Jesse
Shapiro, Carolyn Stein, Michael Tushman, John Walsh, and seminar participants
at the BSE Summer Forum, DRUID, JOMT, NBER Productivity Seminar, NBER Summer
Institute, The Ohio State University, Northwestern University, WEFI, and REER.

###### Abstract

The design of research grants has been hypothesized to be a useful tool for
influencing researchers and their science. We test this by conducting two
thought experiments in a nationally representative survey of academic
researchers. First, we offer participants a hypothetical grant with randomized
attributes and ask how the grant would influence their research strategy.
Longer grants increase researchers’ willingness to take risks, but only among
tenured professors, which suggests that job security and grant duration are
complements. Both longer and larger grants reduce researchers’ focus on speed,
which suggests a significant amount of racing in science is in pursuit of
resources. But along these and other strategic dimensions, the effect of grant
design is small. Second, we identify researchers’ indifference between the two
grant design parameters and find they are very unwilling to trade off the
amount of funding a grant provides in order to extend the duration of the
grant — money is much more valuable than time. Heterogeneity in this
preference can be explained with a straightforward model of researchers’
utility. Overall, our results suggest that the design of research grants is
more relevant to selection effects on the composition of researchers pursuing
funding, as opposed to having large treatment effects on the strategies of
researchers that receive funding.

##  1 Introduction

Firms, governments, and philanthropies that fund science must decide not only
how much to invest in researchers, but how to structure those investments.
These funders often invest in science invest in science in the form of grants,
which award researchers a fixed amount of money for a fixed amount of
time.111Research grants are “ _upfront payments for the delivery of
incompletely specified and non-contractable R &D output_” (Azoulay and Li,
2021; pg. 120); see Azoulay and Li, (2021) for a detailed review. Grants are
explicitly used in academic science, and often implicitly in for-profit
science when firms pre-commit some amount of time and capital to an uncertain
R&D project (Kerr and Nanda, 2015). Many funders express an interest in using
these parameters of money and time to incentivize more socially valuable
science. For example, the prestigious Howard Hughes Medical Institute recently
extended their grants from five to seven years, claiming that those “ _years
of stable support allows [researchers] to take more risk and achieve more
transformative advances._ ”222See the following links for more on this and
other funding institutions proclaiming the importance of grant design: Howard
Hughes Medical Institute, National Institutes of Health, Wellcome Trust.

This argument — the design of grants can influence researchers’ strategies —
has roots in incentive-based theories of innovation and venture
financing.333For example, Holmstrom, (1989) and Manso, (2011) have potential
implications for the optimal structure of investments in research. We discuss
other related work below. Different grant designs might induce researchers to,
for example, acquire different inputs, plan over different time horizons, or
diversify their resources across different types of projects. However, the
practical relevance and specific implications of any particular theory remains
unclear given the backdrop of institutions and incentives that researchers
operate amongst (Dasgupta and David, 1994; Stephan, 1996) and the rarity of
natural experiments in this setting.

Understanding the usefulness of grant design as a policy lever is further
complicated by the fact that researchers usually choose what grants to pursue.
Thus, funders must understand researchers’ preferences and incorporate this
knowledge of demand when choosing which grants to supply. For example, in 2015
the National Institutes of Health (NIH) introduced a new grant structure that
offered more time, but less money, in hopes of encouraging researchers “ _to
pursue new research directions as opportunities arise_ ” and “ _avoid abrupt
termination of laboratory support_.” Ideally, the NIH would have known
researchers’ preferences over money and time when they designed this program,
but this information is hard to elicit and quantify; the market for science is
largely devoid of the prices and variation in attributes that can be used to
identify specifics of demand.444 See here for more on this particular grant,
referred to as the “MIRA” mechanism, and here for more on researchers’ mixed
responses to the money-time tradeoff that MIRA required of participants. So
while the potential for distortions in researchers’ strategies has long been
appreciated (Nelson, 1959; Arrow, 1962), we still have a limited empirical
understanding of researchers’ preferences over grant attributes and how these
attributes could be used to influence the direction of science; what are the
treatment and selection effects induced by research grant designs?

In this paper, we identify researchers’ preferences and beliefs over grant
designs using a nationally representative sample of research-active professors
spanning all major fields of science across all major academic research
institutions in the US (Myers et al., 2023). We focus on two specific
empirical questions: (1) how would researchers change their strategies if they
were awarded grants of different funding amount (dollars available) and
duration (years those dollars are available); and (2) what are researchers’
preferences over grant designs; how willing are they to trade off money and
time? Of course, all researchers want more money and more time, but we are
able to identify the _relative_ value researchers place on these two
attributes and their usefulness in managing the rate and direction of
innovation. Funders have attempted to solicit this sort of preference before
(National Science Foundation, 2002; Royal Society of New Zealand, 2004), but
not in a systematic way tied to an economic model as we do here.

Our first empirical exercise studies how effective grant design could be in
influencing researchers’ strategies. We ask researchers what strategic changes
they would undertake if they received a hypothetical grant, the amount and
duration of which is randomized. This hypothetical grant is posed as a sudden
influx of funding, and we provide specific details to eliminate idiosyncratic
features of grants that might contaminate researchers’ responses.555For
example, we emphasize that the funding may be used for any research project
and the funding amount and duration are fixed and non-negotiable in order to
rule out no-cost extensions. This sort of unanticipated grant receipt is rare
in practice. However, it allows us to isolate researchers’ responses to grant
designs without any selection or competitive effects driven by researchers’
endogenous choices (which would have occurred if we instead posed the thought
experiment about funding competitions).

Researchers respond by choosing from a menu of five subjective options, which
are described in discipline-agnostic language, as to how the randomized grant
would lead them to change some strategic aspects of their
work.666Specifically, the five options are: “Pursue riskier projects”,
“Increase speed”, “Pursue projects less related to your current work”,
“Increase the size of ongoing projects”, and “Increase accuracy or
reliability”. We do not take a stance on the relative social value of any
strategy. Instead our goal is to inform a policymaker or manager who already
has preferences over these strategies. “Strategy” is a nebulous concept in the
context of basic science and our formulations are not clearly mutually
exclusive. But they are informed by extensive interviews with researchers
across all major fields in the sample. Furthermore, the summary statistics and
results of the experiment are consistent with researchers treating these
options as relatively exclusive. Two options are based on the “risk-versus-
speed” trade-off at the core of principal-agent models that seek to balance
risk-taking against shirking with the use of long-term incentives (e.g.,
Holmstrom, 1989; Manso, 2011; Lerner and Wulf, 2007; Aghion et al., 2013; Tian
and Wang, 2014; Nanda and Rhodes-Kropf, 2017). Two options are based on the
“explore-versus-exploit” trade-off vis-à-vis the role of financial constraints
in shaping innovators’ willingness to pursue novel and uncertain opportunities
(e.g., Froot et al., 1993; Hall, 2002; Brown et al., 2009; Hottenrott and
Peters, 2012; Krieger et al., 2022). Lastly, one option focuses on
researchers’ willingness to engage in activities that improve the accuracy or
reliability of their work, which is motivated by the growing attention placed
on so-called replication crises that may be arising because of constraints on
researchers time horizons or budgets (e.g., Pashler and Harris, 2012; Smaldino
and McElreath, 2016; Kasy, 2021).777We also ask researchers how they would
spend the grant funds across a menu of input categories to investigate how
certain inputs are connected to certain strategies.

Through a combination of standard regressions and machine-learning methods
that search for heterogeneous treatment effects, we find two main results in
this experiment. First, we find that longer grants increase researchers’
willingness to pursue riskier projects, but only amongst tenured professors.
On the other hand, we find that all researchers are more likely to decrease
the speed of their work when receiving longer grants. This result is
consistent with theories such as Holmstrom, (1989) and Manso, (2011), where
long-term incentives (here, in the form of longer periods of guaranteed access
to funding) can induce risk-taking. However, it suggests that grant duration
is only useful as an incentive for risk-taking amongst researchers who have
long-term job security and/or a sufficient stock of reputation and resources
with which they are willing to gamble. Furthermore, these findings emphasize
the potential trade-offs involved with the provision of long-term incentives
(i.e., Nanda and Rhodes-Kropf, 2017).

We also find a connection between grant funding amounts and researchers’
explore-versus-exploit decisions. Researchers receiving larger grants are more
likely to choose the exploit-oriented option, while those receiving smaller
grants are more likely to chose the explore-oriented option. At a surface
level, this result stands in contrast to firms’ behaviors when they experience
cash windfalls, where larger windfalls induce more exploration (Krieger et
al., 2022). We propose two alternative ways of rationalizing this result,
which has novel implications for grant designs and echoes Wu et al.,’s (2019)
finding that small research teams (who presumably have less funding) are
responsible for a disproportionate share of scientific “disruptions”.888This
result is also in line with theories and some empirical evidence from the
strategic management and social-psychology literatures that resource
constraints can foster creativity or more entrepreneurial thinking (e.g.,
Starr and MacMillan, 1990; Moreau and Dahl, 2005; Hoegl et al., 2008;
Scopelliti et al., 2014).

We find no connection between grant designs and researchers’ willingness to
focus more on the accuracy or reliability of their work. To the extent this
sort of effort is underincentivized by existing structures of science, it does
not appear that marginal changes to grant designs would be an effective
countermeasure.

In all cases, the magnitudes we identify, when scaled to match the variation
we observe in real grant designs, indicate that the effects are small.
Realistically-sized changes to grant designs (e.g., in the range of 15–30%)
correspond to only 1–2 percentage point changes in the probability of
researchers choosing certain strategies. Notably, our investigations into the
potential for heterogeneous treatment effects rarely yields any statistically
significant evidence of heterogeneity. Together, this suggests that the
practical influence of grant design on researchers’ strategies may be quite
limited.

Our second empirical exercise is focused on understanding researchers’ demand
for different types of grants. We use another thought experiment to estimate
researchers’ willingness to trade off funding amount and funding duration. We
assume that each researcher $i$’s indirect utility from a grant of amount $A$
and duration $D$ is of the form $v_{i}(A,D)=\alpha_{i}A^{\gamma}D^{1-\gamma}$,
where the key parameter is $\gamma\in[0,1]$. A smaller $\gamma$ would indicate
a higher willingness to trade off funding amount for duration and reflect a
belief that time is an important constraint on researchers’ ability to
leverage funding.

We estimate that the average researcher is willing to trade off approximately
$50,000 from their grant to extend it by 1 year ($\gamma\approx 0.8$). This
also implies that a 1% increase in grant size is valued nearly four times more
than a 1% increase in grant duration — relatively speaking, grant size is much
more important to researchers than duration. These estimates provide funders
with a more concrete sense of researchers’ demand for different grant types
and suggest that researchers do not view the duration of any single grant as
an important constraint on their research pursuits given their preferences,
incentives, and expected access to future funding sources.999Of course, this
doesn’t imply that researchers’ preferences are perfectly aligned with the
social optimum.

We also test the extent to which researchers have heterogeneous preferences
for money and time. We motivate these tests with a simple model which predicts
that we should see the strongest preference for money, relative to time,
amongst researchers who are (1) more capital-intensive, (2) can more easily
access funding through other sources, (3) are less risk-averse, (4) have
higher discount rates, and (5) receive more direct utility from research grant
funding (e.g., in the form of salary buyouts). When we split the sample along
each of these dimensions using direct or proxy measures, we find results
consistent with our predictions. Most importantly, these results show that
manipulating grant design to change strategies will also induce selection
effects that will generate changes in the composition of researchers being
funded.

The most closely related paper to ours is Azoulay et al., (2011), which
provides evidence that grant design may be instrumental. They compare the
publication output of premiere biomedical scientists who receive research
grants awarded by either the Howard Hughes Medical Institute (HHMI) or the
NIH. In short, the HHMI grants are characterized by being larger, longer, and
generally having fewer constraints. Azoulay et al., (2011) find that HHMI
awardees produce more high-impact papers _and_ more low-impact papers, and are
more likely to change the direction of their science. Another closely related
study is Wang et al., (2018), who investigate output differences for
researchers funded under grant structures that vary in their competitiveness.
However, in both cases, understanding the generalizability of the results and
attributing the difference in outcomes to the specific aspects of the grants
is difficult. Our work builds on theirs by clearly separating the selection
and treatment effects of grant design, while also exploring the full
population of academic science.

Other relevant literature includes evaluations of researchers’ preferences
over job attributes (e.g., Stern, 2004; Roach and Sauermann, 2010; Sauermann
and Roach, 2014), life-cycle effects (e.g., Levin and Stephan, 1991), risk-
taking in science (e.g., Mandler, 2017; Franzoni and Stephan, 2022; Veugelers
et al., 2022; Clancy, 2023), how the novelty of an idea shapes researchers’
beliefs (e.g., Boudreau et al., 2016; Wang et al., 2017), and the role of
speed and priority in science (e.g., Merton, 1957; Hagstrom, 1974; Smaldino
and McElreath, 2016; Hill and Stein, 2019). Few studies have been able to
study researchers’ demand for research funding specifically (e.g., Myers,
2020), although a number of studies have made progress in understanding how
researchers react to supply shocks (e.g., Tham, 2023; Tham et al., 2023) and
how certain institutions can facilitate the diffusion of research inputs
(e.g., Furman and Stern, 2011; Furman et al., 2012; Agrawal et al., 2016;
Murray et al., 2016; Teodoridis, 2018).

Our approach follows a long line of prior work that has used surveys of
researchers to uncover their otherwise unobservable features, choices,
preferences, or beliefs (e.g., Levin and Stephan, 1991; Fox and Stephan, 2001;
Stern, 2004; Sauermann and Cohen, 2010; Curty et al., 2017; Levecque et al.,
2017; Shortlidge and Eddy, 2018; Cohen et al., 2020; Philipps, 2022). Ideally,
we would pursue our questions with a field experiment or variation induced by
a natural experiment. However, a field experiment for our specific questions
would be prohibitively costly — the median observed grant at institutions in
our population is $300,000 and 3 years long. And, as we show in the next
section, the distribution of observed grant designs is quite concentrated as
funders are far from fully exploring the space of feasible grant designs.
Thus, useful natural experiments with sufficient statistical power are
virtually non-existent. So while our thought experiments involve hypothetical
scenarios, we are able to explore grant designs that are plausible but rarely
seen in practice. While some of our measures are subjective, they provide a
unique, ex-ante observation of researchers’ beliefs in contrast to the typical
approach in this literature, which relies on censored, ex-post observations
(e.g., transformations of publication output).101010See Stantcheva, (2022) for
a full discussion of the trade-offs of survey-based empirical analyses.

Before detailing the experiments, we pay considerable attention to the
respondents’ attention and the potential for non-response bias. As a test of
attention, we compare researchers’ self-reported annual salaries to those we
observe on publicly available websites (only for researchers at institutions
where such data is made public). We find strong agreement between the two,
which suggests mostly truthful reporting along this dimension.

Our approach to non-response involves two parts. First, we use data obtained
for both respondents and non-respondents on the amount of research funding
flowing to their institutions (sourced from the National Science Foundation)
and, for a subset of the population, their publication, and grant flows. In
many dimensions, we do estimate that respondents are significantly different
statistically speaking; however, these differences are all very small
economically speaking, typically close to 5%.

Our second approach to potential non-response bias makes use of the randomized
participation incentives and reminders in the survey. Both the incentives and
reminders had a significant influence on researchers’ willingness to complete
the survey, which provides us with the variation necessary to implement the
sample selection correction method of Heckman, (1979).

The paper proceeds as follows: Section 2 provides motivating summary
statistics about the current distribution of research grants; Section 3
details the survey, including the population, sampling process, the survey
instrument, and our approaches to testing for, and mitigating the potential
of, non-response bias; Section 4 describes the first thought experiment, which
tests how grant designs influence researchers’ strategies; Section 5 describes
the second thought experiment, which solicits researchers’ preferences over
grant attributes; lastly, Section 6 discusses our results in the context of
related work and motivates future studies on policy tools for managing the
rate and direction of innovation.

##  2 Distribution of current grant designs

Historically, the distribution of grants flowing to universities has been
difficult to observe. Collecting and connecting data from the dozens of large
government agencies and hundreds of smaller organizations is no small task,
and surveying researchers about their full history of grants is not
reasonable. Thankfully, the “Dimensions” data set provides a new look at what
types of grants researchers are receiving by aggregating across a large number
of funders who publicly report data (Digital Science, 2018).111111As of this
paper, Dimensions sources grant data from 233 different US-based funders.
Because it relies on publicly reported grants, the Dimensions data is limited
in its coverage of research investments at universities by businesses, so here
we can focus only on non-business grants.121212Recent federal survey evidence
suggests that business-originated research funding accounts for roughly 5% of
all annual research funding at institutions of higher education (National
Science Foundation, 2023).

Figure 1 provide a new look at the distribution of grant designs across U.S.
academic research institutions. It shows the joint distribution of the size
and duration of research grants. For consistency, we restrict our attention to
grants awarded to the institutions from which we sourced our survey
population, which we detail in the next section. In short, this includes the
150 largest institutions of higher education in the US per the annual amount
of R&D funding flowing to the institutions.We start with all 311,477 grants in
the Dimensions data from 2000–2012 awarded to any researcher at all
institutions in our population. We then restrict our attention to grants from
US-based funders (295,004) with funding amount data (259,358). Of these,
222,041 are within the support of our thought experiments: 10 years long or
less, and $2 million or less. 131313We focus on 2000–2012 to avoid censoring
since duration is measured ex-post in this data.

Figure 1: Distribution of research grant designs

> _Note_ : Based on 222,041 grants at the focal institutions from 2000–2012.
> Data sourced from the Dimensions database (Digital Science, 2018).

A few patterns emerge. Grants longer than 6 years are exceptionally rare. Even
small, long grants that might resemble some form of insurance are rare; less
than 1.5% of grants are longer than 6 years and smaller than $1 million.
Another clear pattern is the positive correlation between the size and
duration of research grants. On average, a grant that is one year longer will
award about $500,000 more in total.141414This also holds on a per-year basis;
an additional year is associated with $40,000 more per year.

Overall, it is clear that the full space of reasonable grant designs has not
yet been explored much. The patterns illustrated here may reflect some optimal
staging of investments by funders looking to learn information about risky
projects (Dixit and Pindyck, 1994). However, it may also reflect a missing
market for certain grants, possibly driven by the fiscal costs of long-term
outlays or rules and regulations that restrict the duration of financial
arrangements between funders and grantees.151515For example, see here for the
NIH’s policy that limits grant “segments” to a maximum of 5 years.

Below, we use our thought experiments to understand what might happen to the
type of science and composition of researchers if funders start to change this
distribution. Also, we use this observed data to make some novel inferences
about funders’ preferences over grant designs and compare those to our
estimates of researchers’ preferences.

##  3 Survey, population, and sample statistics

###  3.1 Population and sampling

We make use of the National Survey of Academic Researchers, which is detailed
in Myers et al., (2023). We provide a brief overview of the survey methodology
here. The target population is US professors who conduct research at major
institutions of higher education. This was formalized by selecting
approximately 150 of the largest institutions in the US per their total R&D
funding reported in the National Science Foundation’s 2019 Higher Education
R&D (HERD) survey (National Science Foundation, 2023) and manually collecting
professors’ information from public websites. Recruitment e-mails were sent to
a random 50% of the population from October 2022 to March 2023.

The population consisted of 264,036 unique e-mails. We e-mailed a total of
131,672 individuals and 4,388 (3.33%) completed the survey.161616This response
rate is more than twice what has been obtained from sourcing academic
researcher contacts from the corresponding author data contained within the
publication record (e.g., Myers et al., 2020). We then restrict the sample to
the 4,175 individuals (95.1% of respondents) who reported being a professor
and spending a non-zero amount of time on research. Our final sample consists
of professors from engineering, math, and related fields (703), humanities and
related fields (786; e.g., history, linguistics), medical sciences (1,165;
e.g., schools of medicine or public health), natural sciences (655; e.g.,
biology, chemistry, physics), and social sciences (866; e.g., economics,
psychology, sociology). We use these four aggregate groupings of fields
throughout our discussion and empirical analyses given the small sample sizes
within narrower field definitions. Table 1 reports some key summary statistics
for our sample. The summary statistics related to the thought experiments are
reported in the sections below, and Appendix A.1 contains a larger table with
all covariates in the survey used in any of our analyses.

Table 1: Summary statistics — survey sample

| count | mean | sd
---|---|---|---
_Broad field_ {0,1} |  |  |
Engineering, math, & related | 4,175 | 0.17 | 0.37
Humanities & related | 4,175 | 0.19 | 0.39
Medical & health sciences | 4,175 | 0.28 | 0.45
Natural sciences | 4,175 | 0.16 | 0.36
_Research_ |  |  |
Guaranteed funding | 4,175 | 430,699.40 | 1,085,486.21
Fundraising expectations | 4,175 | 549,238.32 | 1,055,926.36
_Professional_ |  |  |
Has tenure {0,1} | 4,175 | 0.59 | 0.49
Total salary | 4,175 | 156,820.36 | 93,741.46
Soft-money salary | 4,175 | 25,336.53 | 45,407.18
Total work hrs./week | 4,175 | 49.88 | 13.06
Share of time spent on research [0,1] | 4,175 | 0.39 | 0.20
Share of time spent on fundraising [0,1] | 4,175 | 0.09 | 0.11
_Socio-demographic_ |  |  |
Age | 4,084 | 49.31 | 12.52
Female {0,1} | 3,992 | 0.41 | 0.49
Non-white race/ethnicity {0,1} | 4,110 | 0.24 | 0.43
Num. dependents in household | 4,104 | 0.98 | 1.13
U.S. immigrant {0,1} | 4,088 | 0.27 | 0.44

> _Note_ : “Guaranteed” research funding includes funds from prior awards and
> any other guaranteed sources over the next 5 years; “Fundraising
> expectations” are researchers’ beliefs about how much funds they will obtain
> from non-guaranteed sources over the next 5 years. All variables are
> continuous and bounded below at zero unless otherwise specified. Responses
> to socio-demographic questions were not mandatory, hence the differences in
> the count of observations.

It is important to demonstrate that survey participants have some demand for
funding for their research since the experiments are irrelevant to any
researcher with no demand for funding (e.g., their only input is their own
effort). To illustrate this, Appendix Figure A1 plots the distribution of
expected research funding over the five years from the time of survey forward
for each of the four broad scientific fields. This is the sum of guaranteed
funding the researcher has from either prior awards or guaranteed future
funding flows plus their expected fundraising over the same period. The vast
majority ($>$75%) of researchers in each field have non-zero funding
expectations. Furthermore, each distribution shows significant support over
the range studied in the experiments below; research grants are important
across all of science.

###  3.2 Addressing potential survey biases

#### Attention and representativeness

Ideally, our respondents would report all answers truthfully and these
responses would reflect the full population in terms of their preferences
over, and responses to, different grant designs. We can never formally test
this, but we can take some steps to investigate the possibility of inattention
and non-response bias and, in the case of non-response bias, possibly account
for it.

As a test of respondents’ attention and their willingness to report
truthfully, Myers et al., (2023) compares respondents’ self-reported salaries
to their publicly-reported salaries for the subset of researchers at
institutions that make such data public. The two show a close degree of
alignment with a correlation of about 0.75, and the difference between the
self- and publicly-reported salary is less than 30% for roughly three-quarters
of observations.171717Discrepancies between self- and publicly-reported
salaries can be due to a combination of recall error, inattention, or the time
lag between the publicly-reported salary and when the survey was taken. This
suggests the vast majority of respondents are responding truthfully along this
dimension.

Myers et al., (2023) also reports two comparisons of the population and sample
professors using observable data on institutional funding levels and
professor-level grant funding and publication output. In both cases, the
respondent sample is quite similar to the population. The average difference
in funding amounts between the institutions of respondents and non-respondents
is generally in the range of 4–6%, and there are no statistically significant
differences in the grant funding or publication output metrics between the two
groups. Some of these exercises are replicated in Appendix A.2Overall, our
respondent sample appears very similar to the population along many observable
dimensions.

#### Sample selection correction

Despite the overlap in observable data, our respondents may still differ on
other unobservable dimensions. If these unobservable differences are related
to our focal parameters of interest (e.g., researchers preferences over grant
designs), this may generate a non-response bias. To account for this
possibility, we implement the sample selection correction approach first
developed by Heckman, (1979). This approach makes use of excluded variables
that cause entry into the sample (i.e., completion of our survey), but are
plausibly orthogonal to the parameters of interest.

We generated the variation necessary for this approach with randomized
recruitment strategies. During the recruitment process, we randomly allocated
each e-mail to one of four different incentive arms and one of three different
reminder arms. The four incentive arms were: (1) no incentive, (2) entry into
a lottery to win a gift card, (3) the ability to vote for a set of charities
to receive a donation, and (4) both the second and third incentives. The three
reminder arms involved no, one, or two reminders, respectively.

Appendix A.3 reports the effect of the incentives and reminders. We find the
stronger incentives lead to a small increase in completion rates, while the
reminders are much more powerful. Both induce a significant increase in
completion rates; the completion rate for researchers that receive no
incentives and no reminders is roughly 1.9% whereas the completion rate for
those who receive both incentives and both reminders is roughly 4.5%.

Using the randomized variation from these incentives, combined with the
information on each professor’s rank, institution, and inferred field of
study, we then estimated individuals’ propensity to complete the survey
conditional on being sampled. From these predicted probabilities, we
constructed the inverse mills ratio and include this variable in all
regressions. Assuming that that researchers responsiveness to either
participation instrument is orthogonal to a focal parameters, this will
control for any unobservable differences (in a focal parameter) between our
sample and the population (Heckman, 1979).181818Obviously, this approach does
sacrifice some potential sample size. But using the full set of reminders and
incentives in all recruitments would have eliminated our ability to implement
any correction for potential non-response bias.

##  4 Grant design and research strategy

###  4.1 Theoretical framework

How might grant design affect researchers’ strategies? To begin, it is useful
to map the two focal grant design parameters, money and time, to the two
levers that much of the literature on managing innovation has focused,
financial slack and long-term incentives.

In many contexts, financial slack is often operationalized as free cash flow.
Here, in the setting of academic research grants, the clear analogue is the
amount of funding. Larger grants can enable the acquisition of more scientific
inputs because funding for one’s research is quite costly to obtain. Academic
researchers spend an average of about 15% of their time fundraising for their
work (Myers et al., 2020), and success rates at major funding agencies are on
the scale of 10–20% (De Vrieze, 2017).

Long-term incentives are often operationalized as the duration of employment
contracts, the time-horizon of stock options, or the use of “golden
parachutes”. Here, the analogue is the duration that grant funds are made
available. Longer grants can change researchers’ expectations and uncertainty
about their future funding, which may have important consequences. For
example, Tham, (2023) and Tham et al., (2023) show how interruptions to
researchers’ funding expectations can disrupt their spending flows and their
ability to retain skilled labor in their laboratories.

In what follows, we outline the strategic dimensions along which these
parameters of grant design may be instrumental. We don’t take a stance on the
social value of any of these dimensions or the degree to which the private and
social value may diverge. Rather, our goal with the following experiment is to
understand the extent to which policymakers or managers might incentivize
particular strategies with particular grant designs.

Long-term incentives, risk-taking, and speed. Principal-agent and multi-
tasking models of innovation predict that long-term reward structures can
induce risk-taking (i.e., increasing the expected variance of outcomes) at the
potential cost of shirking or other forms of negative selection (Holmstrom,
1989; Manso, 2011; Hellmann and Thiele, 2011; Nanda and Rhodes-Kropf, 2017).
This prediction has been confirmed in laboratory experiments (Ederer and
Manso, 2013) and observed in natural experiments amongst for-profit firms in
R&D-intensive sectors (Lerner and Wulf, 2007; Aghion et al., 2013; Tian and
Wang, 2014). The results of Azoulay et al., (2011) are consistent with this
prediction where HHMI-funded scientists (who have longer guaranteed funding
arrangements) produce more very-high and very-low cited publications, which is
evidence of a riskier strategy. The extent to which longer grants might induce
risk-taking compared to reducing the speed with which researchers work is
unclear.

Financial slack, exploration, and exploitation. Individuals and organizations
that are financially constrained and face costly external finance are expected
to invest less in novel opportunities that are less related to their prior
pursuits (Hall, 2002; Brown et al., 2009; Hottenrott and Peters, 2012). A
recent, clear example of the corollary is Krieger et al., (2022) who show that
when pharmaceutical firms experience cash windfalls they are more likely to
explore novel molecular therapies. Azoulay et al., (2011) is also consistent
with this prediction, finding that the HHMI-funded scientists (who receive
larger funding amounts) are more likely to change the direction of their
research. But the extent to which this could be attributed to their funding
levels per se is unclear.

Constraints and replicability. Models of racing in science suggest that the
priority reward structure has the potential to incentivize low-quality
pursuits of high-value research questions (Bryan and Lemus, 2017). This sort
of competitive pressure appears to be relevant in practice, especially when
researchers are financially- or time-constrained, leading to replication
concerns, biased statistical reporting, and low quality experimental designs
(Pashler and Harris, 2012; Smaldino and McElreath, 2016; Andrews and Kasy,
2019; Hill and Stein, 2021; Bryan et al., 2022). There continue to be numerous
calls for new institutions and incentives to address these issues,191919For
example, see this, this, or this commentary and coverage. but whether grants
with certain designs may be able to counteract this pressure remain unclear.

There is certainly heterogeneity across professors in terms of the relevance
of grants for their work, as well as the institutions and incentives that may
modulate the effect of grant designs. In this vein, the thought experiment is
designed to be as discipline-agnostic as possible, and our analyses make full
use of modern machine learning techniques to search for heterogeneous
treatment effects.

###  4.2 Experimental design and summary statistics

We ask researchers how they would change their research strategies if they
received some hypothetical grant, the size and duration of which is
randomized. The value and duration of this hypothetical grant are randomized
over {$100,000, $250,000, $500,000, $1,000,000, $2,000,000} and {2, 3, 4, 5,
6, 7, 8, 9, 10} years, respectively.

If grant attributes can be used to modulate the scientific production
function, we should see researchers systematically changing their strategies
when receiving grants with different designs. To capture this, we present
researchers with five possibilities of how they might change their strategies:
“Pursue riskier projects”, “Increase speed”, “Pursue projects less related to
your current work”, “Increase the size of ongoing projects”, and “Increase
accuracy or reliability”. We ask them to select the two most important changes
the grant would enable them to make in their research, requiring two choices
to be made.

The first two options (“Pursue riskier projects”, “Increase speed”) are
motivated by the potential relationship between risk-taking (i.e., increasing
the expected variance of outcomes) and pace-of-work underlying the
aforementioned principal-agent models of innovation. The next options (“Pursue
projects less related to your current work”, “Increase the size of ongoing
projects”) are motivated by the potential explore-versus-exploit decisions
underlying the aforementioned models of financial constraints. The last option
(“Increase accuracy or reliability”) is motivated by the fact that, while
problems with reliability and replication in science have become increasingly
publicized, the extent to which policy levers like grant designs might be able
to address this problem remains unclear.202020We lack any ground truth with
which to validate the descriptions of these strategies; however, their
phrasing was informed by numerous pilot interviews with researchers across a
variety of fields.

Validated, ex-ante measures of strategy in the context of science do not yet
exist, but each of these options are clearly motivated by prior studies of
researchers. The obvious limitation here is that these are subjective
descriptions of a subset of all possible strategies researchers could
undertake. Still, the traditional approach of making inferences from
researchers’ output (e.g., publications) have their own set of
drawbacks,212121Those measures are ex-post realizations and may suffer from a
truncation bias when failed projects do not produce observable output. For
example, see Franzoni and Stephan, (2022) for a discussion of the difficulties
of quantifying risk-taking in science. and so we view this approach as a new,
complementary one.

Table 2 reports the summary statistics for the randomized and response
variables in this thought experiment.

Table 2: Summary statistics – strategy thought experiment

| count | mean | sd | min | max
---|---|---|---|---|---
_Randomized elements_ |  |  |  |  |
Grant duration (years) | 4,175 | 6.06 | 2.56 | 2 | 10
Grant amount ($) | 4,175 | 778,239.52 | 684,075.23 | 100,000 | 2,000,000
_Strategic changes{0,1}_ |  |  |  |  |
Increase speed | 4,175 | 0.36 | 0.48 | 0 | 1
Pursue riskier projects | 4,175 | 0.53 | 0.50 | 0 | 1
Pursue new directions | 4,175 | 0.33 | 0.47 | 0 | 1
Increase size of ongoing projects | 4,175 | 0.61 | 0.49 | 0 | 1
Increase accuracy | 4,175 | 0.17 | 0.38 | 0 | 1

> _Note_ : Summary statistics of the two randomized elements of the research
> strategy thought experiment, and the respondent’s answers to the question of
> “What are the two most important changes this grant would enable you to make
> in your research?”.

###  4.3 Empirical approach

To explore the effect of grant design on researchers’ probability of choosing
a particular strategy, we estimate five regressions of the form:

$Y^{k}_{i}=h^{k}(A_{i},D_{i},\mathbf{X}_{i})\,\,,$ (1)

where the $Y$ is an indicator that equals one if strategy $k$ is chosen by the
researcher in response to the randomized grant, the funding amount and
duration are $A$ and $D$, and $\mathbf{X}$ is a vector of covariates solicited
elsewhere in the survey (see Appendix A for a full listing of these covariates
and their summary statistics). We explore a range of functional forms of $h$.
In the simplest scenario, we test only the main effects of the randomized
grant amount and duration and estimate five separate OLS regressions using
each of the five indicators as the dependent variable. In the appendix, we
also report results from OLS regressions after using Lasso to select controls
from the rest of the survey data to possibly improve our precision as well as
estimates from a single discrete choice model that jointly estimates how
researchers’ propensity to choose a particular strategy depends on grant
attributes.

Because we force researchers to choose two of the strategic options, our
analyses here are all focused on _relative_ shifts in the mix of strategies
that researchers pursue after receiving a grant. That is, we are not
estimating how researchers’ total scientific output (e.g., publications)
depends on the grant structure. Rather, we assume that researchers are always
undertaking the same fixed “amount of strategy”, and their decision is only
about the mix of strategies to employ — it is purely a directional measure.

For clarity, consider our OLS regression specification:

$Y^{k}_{i}=\alpha^{k}+\beta^{k}_{A}A_{i}+\beta^{k}_{D}D_{i}+\epsilon^{k}_{i}\,\,,$
(2)

which we estimate for each of the five strategies indexed by $k$. Written this
way, our coefficient estimates ($\beta^{k}_{A},\beta^{k}_{D}$) will sum to
zero across the five models (i.e., $\sum_{k}\beta^{k}_{A}=0$); again, we are
only estimating how grant design leads to substitution across these options.
Positive values of our coefficients indicate that, as grants get larger or
longer, researchers increasingly prioritize some strategy because, with those
additional resources, that strategy has become marginally more useful to the
researcher. Conversely, negative values indicate which strategies researchers
substitute away from as grants get larger or longer. Our statistical tests of
the null are therefore tests of whether or not researchers systematically
substitute to or away from particular strategies.

We are particularly interested in the possibility of heterogeneous responses
given the idiosyncrasies of each researchers’ situation. To do this, we use
the causal forests algorithm developed by Wager and Athey, (2018) to estimate
conditional average partial effects (CAPEs) of the grant attributes. This
approach yields informative distributions of effects, but do not directly
reveal the features of researchers associated with larger or smaller CAPEs. To
understand the heterogeneity in a more low-dimensional manner, we then use the
Best Linear Projection methodology to test how the CAPEs vary with a select
number of covariates (Chernozhukov et al., 2018b ).222222All of these analyses
are performed in R using the grf package (Tibshirani et al., 2022).

###  4.4 Results

Table 3 reports our simplest version of the analyses where we conduct five
separate OLS regressions of strategic choice indicators on the grant
attributes.232323Since the independent variables are identical in each
regression, estimating each OLS regression separately is identical to joint
estimation via , for example, seemingly-unrelated-regression approach. We
report conventional robust standard errors as well as the family-wise
$p$-values, which adjust for the fact that we are testing ten hypotheses (five
outcome variables and two independent variables).242424We make use of Jones et
al.,’s (2019) implementation of the free step-down procedure of Westfall and
Young, (1993) as codified in their Stata package wyoung. Appendix B reports
the results from a logit choice model that jointly estimates researchers’
propensities to choose each strategy. It more closely reflects the structure
of the choice facing the respondent, but it yields very similar results.
Likewise, the post-Lasso OLS regressions reported there, where covariates are
included to potentially improve precision, also show very similar results.
Figure 2 reports the results from using causal forests to estimate the
Conditional Average Partial Effect (CAPE) distributions.252525The means of
each CAPE reported in Figure 2 are approximately equal to the mean effects
reported in Table B.

Three patterns emerge, which we investigate and discuss further in the sub-
sections below. First, we find that larger and longer grants lead researchers
to prioritize speed less, but that such grants can also increase researchers’
willingness to take risks. Second, as grant size increases, researchers shift
their strategic focus _towards_ increasing the size of ongoing projects and
_away_ from pursuing work in new directions. In other words, smaller grants
appear to promote exploration and larger grants appear to promote
exploitation. Third, grant designs appear to have no effect on researchers’
focus on the accuracy of their science.

Table 3: Effect of grant designs on researchers’ strategies

|  |  |  |  |  |  | Larger |  |
---|---|---|---|---|---|---|---|---|---
|  |  | More |  | New |  | ongoing |  | More
| Faster |  | risk |  | directions |  | projects |  | accurate
| (1) |  | (2) |  | (3) |  | (4) |  | (5)
log(Duration) | –0.056∗∗∗ |  | 0.012 |  | 0.008 |  | 0.018 |  | 0.018
| (0.015) |  | (0.016) |  | (0.015) |  | (0.015) |  | (0.011)
log(Amount) | –0.025∗∗∗ |  | 0.017∗∗ |  | –0.016∗∗ |  | 0.026∗∗∗ |  | –0.003
| (0.007) |  | (0.007) |  | (0.007) |  | (0.007) |  | (0.006)
Family–wise $p$–values |  |  |  |  |  |  |  |  |
Duration |  $<$0.01 |  | 0.83 |  | 0.85 |  | 0.61 |  | 0.49
Amount |  $<$0.01 |  | 0.12 |  | 0.12 |  |  $<$0.01 |  | 0.85
dep. var. mean | 0.36 |  | 0.53 |  | 0.33 |  | 0.61 |  | 0.17
$N$ obs. | 4,175 |  | 4,175 |  | 4,175 |  | 4,175 |  | 4,175

> _Note_ : Shows results from OLS regressions where the dependent variables
> are indicators that equal one if the strategy listed at the top of the
> column was chosen as one of the “two most important changes” that
> researchers would make in response to receiving a grant of a given
> (randomized) size and duration. Robust standard errors shown in parentheses;
> stars indicate significance levels using the unadjusted $p$-values: ∗
> $p<0.1$, ∗∗ $p<0.05$, ∗∗∗ $p<0.01$. The family-wise $p$-values adjust for
> the ten hypotheses tested and are based on 10,000 bootstraps of the free
> step-down procedure of Westfall and Young, (1993).

Figure 2: Heterogeneous effects of grant designs on researchers’ strategies

(a)

(b)

(c)

(d)

(e)

> _Note_ : Each panel shows the distribution of the Conditional Average
> Partial Effects (CAPEs), which are estimated separately for the two
> randomized grant attributes using the causal forest method of Wager and
> Athey, (2018); the CAPE is defined as $E[cov[W,Y|X]/var[W|X]]$, where
> $W=\\{\text{log(duration), log(size)}\\}$, $Y$ is the dependent variable,
> and $X$ are the covariates. The only two distributions that show
> statistically significant heterogeneity are (1) the effect of grant duration
> on risk-taking ($p$¡$0.001$) in Panel (b), and (2) the effect of grant
> funding amount on undertaking new directions ($p$¡$0.001$) in Panel (c).

#### Speed versus risk

The first column of Table 3, clearly shows that larger and longer grants lead
to an decreased emphasis on speed (equivalently, smaller and shorter grants
lead to an increased emphasis on speed). This result is intuitive, but by no
means guaranteed given the wide array of time horizons over which researchers
are likely optimizing (e.g., due to the heterogeneity in tenure status,
contract lengths, ages, etc.). The socially optimal pace of science is far
from clear given that there are good reasons both that sciences moves “too
fast” (e.g., as in the racing models of Loury, 1979, which has some empirical
support from Bryan et al., 2022), and “too slow” (e.g., as is emphasized by
Nanda and Rhodes-Kropf, 2017 who study how the provision of long-term
incentives affects the extensive margin of innovation). Still, it does suggest
that grant design is a tool for manipulating researchers’ speed.

The large body of work showing how long-term incentives can induce risk-taking
in firms (Lerner and Wulf, 2007; Aghion et al., 2013; Tian and Wang, 2014) and
the analogous results of Azoulay et al., (2011) suggest that we should find
some increased willingness to take risks when receiving longer grants. But we
do not see an average effect of this sort (see Column 2 of Table 3).

However, the CAPE distribution in Figure 2 Panel (b) clearly shows a large
degree of heterogeneity in researchers’ responsiveness to grant duration with
respect to risk-taking. This particular relationship is one of only two CAPE
distributions that represent statistically significant degrees of
heterogeneity. To better understand this heterogeneity, Appendix B.3 reports
the Best Linear Predictors of the CAPEs. The best predictor of researchers’
responsiveness to grant duration with respect to risk-taking is their tenure
status — the effect of a log-point increase in grant duration on researchers’
willingness to take risks is roughly 10 percentage points larger for tenured
professors compared to nontenured professors (compared to statistically
insignificant mean effect of 0.01). This suggests that grant duration may be a
useful incentive for risk-taking as the aforementioned theories would suggest,
but only amongst researchers who have the complementary job security,
reputation, or resources associated with tenure. Conversely, nontenured
researchers may be facing performance evaluations on timescales that render
marginal changes in grant duration irrelevant.

This result is consistent with Azoulay et al., (2011), in that their sample
consists entirely of elite researchers who likely already have, or are clearly
on a path, to a tenured or de-facto tenured positions but are still on the
younger end of the age spectrum. Our findings help guide the generalizability
of those results and identify the conditions under which researchers might be
incentivized to take risks with longer grants.

#### Exploration versus exploitation

Prior work focusing primarily on science in for-profit firms would suggest
that an important mechanism for shifting researchers’ decisions along the
explore-exploit axis would be the size of the grant. Specifically, empirical
studies have shown that when firms’ access to capital increases or they
receive a windfall of cash, they undertake more exploratory research
trajectories in new directions (Krieger et al., 2022). We find something close
to the opposite.

In our setting, larger grants lead researchers to increase the size of ongoing
projects at the expense of moving in new directions — that is, as funding
amounts increase, researchers shift from using those funds to explore to using
them to exploit. Equivalently, as grants get smaller, researchers are more
likely to use those smaller amounts of funds to explore new directions.

There are many ways this could be rationalized. One rationalization would be
that starting new projects involves high fixed costs due to the scarcity of
specialized inputs. That is, if there are certain inputs that researchers need
specifically for the purposes of starting new projects (e.g., a new postdoc to
join their laboratory) and these inputs are scarce enough, researchers may be
forced to commit larger grants to ongoing projects. The net returns to large
investments in new directions may be small because the costs are higher (e.g.,
it is worth spending a small amount of money to explore a new direction, but
scaling this new-direction project quickly becomes expensive because of the
scarce inputs required). We return to this idea below when we examine how
grant designs influence researchers’ choices of inputs to acquire with the
grant.

#### Accuracy and reliability

The least popular strategic choice overall is the _increase accuracy or
reliability_ option, and there are no average or heterogeneous effects to
suggest that grant design can shift researchers’ choices along this dimension.
The importance of preventing and detecting false positives and negatives in
science has come to be appreciated (e.g., Pashler and Harris, 2012; Andrews
and Kasy, 2019). Our results suggest the reason researchers may be
underinvesting in efforts related to accuracy or reliability is _not_ because
they lack the resources to do so, but is because those efforts are
underincentivized in the market for science.

###  4.5 Grants and input allocations

In the preceding analyses, we find that grant design can affect researchers’
strategies (i.e., grant size influencing the choice of new directions versus
larger ongoing projects). How much of this responsiveness is due to the grants
allowing researchers to obtain new inputs for their work? Or, more generally,
how much might grant design influence researchers’ input choices?

A full investigation into such questions is beyond the scope of this paper,
but we did solicit a second set of outcomes in this thought experiment that
allow us to estimate how grants with different designs are used. In the
survey, after respondents were asked to report how their strategies would
change in response to receiving the hypothetical grant, we asked them to
report how they would spend those funds across five different input
categories: (1) their own salary; (2) their own training; (3) travel; (4)
equipment, data, or supplies; and (5) salaries or wages for other new or
current researchers, students, or staff.

Pilot testing with researchers indicated that soliciting precise allocations
across these categories (e.g., percentage of dollars spent) was an excessive
burden and would likely involve a tremendous amount of uncertainty since most
researchers do not have specific knowledge about input costs. As a lower-
burden alternative, we simply asked respondents to indicate if they would
spend “None”, “Some”, or “A lot” of the hypothetical grant funds on each
category. Pilot testing indicated that researchers were much more comfortable
with this format.

To conduct empirical analyses of the responses, we code the three possible
responses to values of 0,1, and 2. Then, we convert these values into shares
by dividing each response by the sum of the responses across all categories
(e.g., if a researcher reports “Some” for all categories, then all categories
will receive a 0.2 share of the funding from the hypothetical grant). As with
our analyses of researchers’ strategic responses, we are less concerned with
the precise magnitudes of the effects and instead our focus is on testing
whether we can reject the null in particular directions.

Table 4: Summary statistics — input allocations

| count | mean | sd
---|---|---|---
_Randomized elements_ |  |  |
Grant duration (years) | 4,175 | 6.06 | 2.56
Grant amount ($) | 4,175 | 778,239.52 | 684,075.23
_Input allocation share [0,1]_ |  |  |
Own salary | 4,175 | 0.18 | 0.13
Own training | 4,175 | 0.08 | 0.10
Travel | 4,175 | 0.18 | 0.10
Equipment or supplies | 4,175 | 0.21 | 0.13
Others’ salaries | 4,175 | 0.34 | 0.14

> _Note_ : Summary statistics of the two randomized elements of the research
> strategy thought experiment (duplicating the first two rows of Table 2) and
> respondents’ allocation of funding amounts across input categories. Input
> allocation responses are standardized such that input shares sum to one.

Table 4 shows the summary statistics for the share of funding that researchers
allocate to each of the five inputs. For simplicity, we report the results
from five separate OLS regressions using the input shares for each category as
the dependent variables in Table 5.262626Unreported results from joint
estimation of all choices via a fractional response regressions with a logit
model for the conditional mean yield very similar results. Overall, grant
duration has no significant effect on input allocations. But we do find grant
size to have some clear effects.

First, we see that larger grants lead researchers to allocate a greater share
of funding towards both their own salary and their own training. By
construction, this increase must come with a decrease elsewhere, and it
appears that almost all of the decrease is occurring in the “Others’ salaries”
category. To be clear, this does not mean that researchers are claiming to
reduce their total spending on this category. Instead, they are indicating
that, as grants get larger in size, a smaller share of the additional funding
will be spent on others’ salaries or wages.

Table 5: Effect of grant designs on researchers’ input allocations

| Own |  | Own |  |  |  | Equipment |  | Others’
---|---|---|---|---|---|---|---|---|---
| salary |  | training |  | Travel |  | or supplies |  | salaries
| (1) |  | (2) |  | (3) |  | (4) |  | (5)
log(Duration) | –0.007∗ |  | 0.000 |  | 0.002 |  | –0.001 |  | 0.005
| (0.004) |  | (0.003) |  | (0.003) |  | (0.004) |  | (0.004)
log(Amount) | 0.008∗∗∗ |  | 0.008∗∗∗ |  | –0.001 |  | –0.003 |  | –0.012∗∗∗
| (0.002) |  | (0.001) |  | (0.002) |  | (0.002) |  | (0.002)
Family–wise $p$–values |  |  |  |  |  |  |  |  |
Duration | 0.37 |  | 0.97 |  | 0.93 |  | 0.97 |  | 0.68
Amount |  $<$0.01 |  |  $<$0.01 |  | 0.97 |  | 0.51 |  |  $<$0.01
dep. var. mean | 0.18 |  | 0.08 |  | 0.18 |  | 0.21 |  | 0.34
$N$ obs. | 4,175 |  | 4,175 |  | 4,175 |  | 4,175 |  | 4,175

> _Note_ : Shows results from OLS regressions where the dependent variables
> are the share of the (randomized) grant that would be allocated to each
> input category. Robust standard errors shown in parentheses; stars indicate
> significance levels using the unadjusted $p$-values: ∗ $p<0.1$, ∗∗ $p<0.05$,
> ∗∗∗ $p<0.01$. The family-wise $p$-values adjust for the ten hypotheses
> tested and are based on 10,000 bootstraps of the free step-down procedure of
> Westfall and Young, (1993).

We cannot test whether this shift in allocations is due to supply constraints
(e.g., researchers cannot attract additional labor to work with them), or
whether this is due to researchers’ demand for labor shrinking as grant size
increases (e.g., their own time is the main constraint that the increased
funding allows them to address).

The fact that we find larger grants increase researchers’ spending shares on
themselves (i.e., via their salary and training) and decrease spending shares
on others (e.g., students, staff scientists) while simultaneously increasing
the probability of increasing the size of ongoing projects while decreasing
their likelihood of pursuing new research directions is quite interesting.
This is consistent with researchers’ teams, their students and staff, being a
key part of helping them explore new research directions.

We lack the variation or data to dig further into this question to disentangle
how much these results are driven by the supply- or demand-side, and we cannot
formally connect specific inputs to specific strategies given the variation in
our data.272727It is tempting to devise a two-stage analyses where we test how
grant design influences input choices, and in turn, how these input choices
effect strategic choices. However, our randomized grant designs are not valid
instrumental variables because grant design may influence other features of
the researchers’ production functions besides these specific input choices
(i.e., we cannot guarantee that the exclusion restriction holds). Still,
recent work has begun to highlight how the wages and employment outcomes of
researchers’ teams can be influenced by funding shocks (Tham et al., 2023),
and continued work on how inputs and strategies are connected within the
scientific production function seems especially promising.

##  5 Researchers’ preferences

###  5.1 Empirical model

The prior analyses focused on the identifying the treatment effect(s) of grant
design conditional on receiving the grant. Here, our focus is on researchers’
preferences over grant designs and how, when given freedom to choose which
grants to pursue, these preferences may lead to selection effects that
influence the composition of researchers who pursue particular grants.

We assume that each grant $g$ is defined by a pair of parameters
($A_{g}$,$D_{g}$), which describe the total dollar amount $A$ of the grant and
the duration $D$ of the grant, in terms of how many years the researcher has
access to the funds. Researcher $i$’s indirect utility from a grant is:

$v_{i}(A_{g},D_{g})=\alpha_{i}A^{\gamma}_{g}D^{1-\gamma}_{g}\,\,,$ (3)

where $\alpha_{i}$ is a researcher-specific taste shifter and $\gamma$ is a
common parameter that describes scientists’ relative preference for funding
amount versus duration.

To motivate the analyses and arrive at our main regression equation, consider
a thought experiment where each researcher $i$ observes a pair of grants
$g=(1,2)$ where ($A_{i1},D_{i1}$) and ($D_{i2}$) are randomly specified by the
experimenter. The respondent is asked to report the value $A_{i2}$ that makes
them indifferent between the two grants such that
$v(A_{i1},D_{i1})=v(A_{i2},D_{i2})$. We assume that researchers’ responses
($\widetilde{A}_{i2}$) are the truth plus i.i.d. noise, such that
$\widetilde{A}_{i2}=A_{i2}+\epsilon_{i}$. This allows us to write:

$\displaystyle\begin{split}\gamma\log(A_{i1})+(1-\gamma)\log(D_{i1})&=\gamma\log(\widetilde{A}_{i2}-\epsilon_{i})+(1-\gamma)\log(D_{i2})\,\,.\end{split}$
(4)

We obtain our main estimating equation by solving Equation 4 for
$\widetilde{A}_{i2}$:

$\widetilde{A}_{i2}=\frac{A_{i1}}{\exp\left(\frac{1-\gamma}{\gamma}\log\left(\frac{D_{i2}}{D_{i1}}\right)\right)}+\epsilon_{i}\,\,,$
(5)

which we estimate via non-linear least squares.282828Note that the design of
this grant is designed to be uncorrelated with the design of the hypothetical
grant in the first thought experiment. Amongst the pairwise correlations of
the two grant attributes in the first thought experiment and the three
attributes in this thought experiment, the absolute value of any correlation
is never larger than 0.03.

Equation 3 is a reduced form representation of a more complex model of
researchers’ production and consumption decisions detailed in Appendix C.
Comparative statics derived from stylized simplifications of that model
illustrate how the $\gamma$ parameter should be related to five key features:
(1) the capital intensity of a researchers’ work (i.e., their returns to scale
with respect to funding), where more capital intensive researchers should have
a larger $\gamma$; (2) the ease with which the researcher can acquire
additional funding through other means (i.e., their fundraising productivity),
where researchers more capable at obtaining funding should have a larger
$\gamma$; (3) researchers’ risk aversion, where more risk-averse researchers
should have a smaller $\gamma$; (4) the discount rate, where researchers that
discount future periods more should have a larger $\gamma$; and (5)
researchers’ direct utility from grant funding (e.g., in the form of salary
buyout), which should be increase in $\gamma$. These are the dimensions along
which we may see heterogeneity across researchers that would signal the
importance of selection effects induced by grant designs.

###  5.2 Thought experiment

To generate the data necessary to estimate Equation 5, we present respondents
with a hypothetical scenario where an anonymous donor would like to award the
researcher with a single research grant. The donor presents the researcher two
options: Grant #1 and Grant #2, where the first is set to be shorter than the
second. The amount of Grant #1 is shown, but the amount of Grant #2 is not.
The respondent is then asked to state the funding amount of Grant #2 that
would make them indifferent between both grants.292929Text in the survey also
indicates to the respondent that all funding associated with the grant would
be awarded immediately, but only could be used within the time-frame
specified. Furthermore, no so-called “no-cost extensions” of the grant would
be considered and all unused funds would expire after the duration of the
grant.

All features shown are randomized. The amount of Grant #1 is a uniform random
draw from {$100,000, $250,000, $500,000, $1,000,000, $2,000,000}. The duration
of Grants #1 and #2 are uniform random draws from {2, 3, 4, 5} and {6, 7, 8,
9, 10} years, respectively. Table 6 reports the summary statistics for the
four key variables related to this thought experiment. The randomized elements
have the expected statistics (e.g., the mean of the variables are
approximately equal the mean of the distributions from which they’re drawn).

Table 6: Summary statistics – grant indifference thought experiment

| count | mean | sd | min | max
---|---|---|---|---|---
_Randommized elements_ |  |  |  |  |
Grant #1: Duration (years) | 4,175 | 3.5 | 1.1 | 2 | 5
Grant #1: Amount ($) | 4,175 | 781,904.2 | 690,313.1 | 100,000 | 2,000,000
Grant #2: Duration (years) | 4,175 | 8.0 | 1.4 | 6 | 10
_Researcher’s response at indifference_ |  |  |  |  |
Grant #2: Amount ($) | 4,175 | 607,129.1 | 584,626.5 | 10,000 | 2,000,000

> _Note_ : Summary statistics of the randomized elements of, and response to,
> the grant preferences thought experiment.

Table 6 also starts to illustrate the degree to which researchers are willing
to trade off funding amount and duration. With Grant #2 being 4.5 years longer
than Grant #1 on average, researchers report that they would be indifferent
between both grants if Grant #2 is roughly $165,000 smaller. This suggests
researchers would be willing to trade approximately $35,000 per year gained on
a grant. Below, we report the marginal rates of substitution based on our
estimation of the formal model of researchers’ utility.

###  5.3 Results

Table 7 reports our estimates of $\gamma$ obtained by estimating Equation 5
via non-linear least squares. In the full sample (Col. 1), we estimate
$\gamma$ to be 0.77, which implies a marginal rate of substitution of
approximately $57,000 per year at the means of the Grant #1 distribution.
Relatively speaking, dollars appear to be three and a half times as important
as duration in the context of grant design.

Table 7: Researchers’ willingness to trade off grant funding amount and
duration

| (1) | (2) | (3) | (4)
---|---|---|---|---
$\gamma$ | 0.767∗∗∗ | 0.779∗∗∗ | 0.817∗∗∗ | 0.848∗∗∗
| (0.00698) | (0.00676) | (0.00592) | (0.00533)
Marginal rate | 57,375 | 53,869 | 42,862 | 34,420
of substitution | [7.3%] | [6.9%] | [5.5%] | [4.4%]
Excl. above ptile. |  | 99th | 95th | 90th
$N$ obs. | 4,175 | 4,134 | 3,978 | 3,777

> _Note_ : Reports results from estimating Eq. 5 using alternative samples;
> larger $\gamma$ values reflect a stronger taste for funding amount compared
> to duration; marginal rate of substitution ($ per year) is reported at the
> means and is shown in brackets as a percentage of average funding amount;
> the “Excl. above ptile.” samples drop responses where
> $(A_{i1}-\widetilde{A}_{i2})/A_{i1}$ is at or above the reported percentile;
> robust standard errors in parentheses; ∗ $p<0.1$, ∗∗ $p<0.05$, ∗∗∗ $p<0.01$.

Columns (2–4) of Table 7 drop small portions of the sample that appear to be
those most willing to trade size for duration in order to test the sensitivity
of our results to outliers. Excluding the 1–10 percent of the most responsive
researchers yields $\gamma$ estimates in the range of 0.78–0.85, which imply
marginal rates of substitution of approximately $53,000-35,000 per additional
year of grant duration. These are small amounts relative to the size of the
grants, roughly 4-7%.

Appendix C.1 includes an alternative version of Table 7 that reports the
coefficient on the sample selection correction control (i.e., the standardized
Inverse Mills Ratio). In all cases the control is not significant at
conventional levels, and the magnitude is small.303030We find that a one
standard deviation increase in the control is associated with a 1% increase in
the respondent’s answer to the thought experiment compared to the sample mean.
This suggests that our sample is not comprised of individuals with especially
high or low $\gamma$ values compared to the population.

To visualize the magnitudes implied by our estimates, Figure 3 plots the
indifference curves over the support of the experimental variation. The plot
illustrates how, for grants less than roughly $500,000, researchers are
relatively unwilling to sacrifice any of those funds to extend the duration of
the grant. Once grant sizes extend beyond $1,000,000, this tradeoff becomes
more meaningful.

Figure 3: Grant design indifference curves

> _Note_ : Plots indifference curves over the support of the variation in the
> duration and size of the hypothetical research grants considered by
> respondents; based on the preferred $\gamma$ estimate (Table 7, Col. 1).

Appendix C.5 reports results from an alternative specification of the
researchers’ utility function that does not assume the constant-returns-to-
scale assumption of Equation 3. This is done by leveraging another component
of the survey, which we detail further in the Appendix. In this alternative
specification, we can estimate a separate parameter for both size and duration
preferences. Still, we find that the relative value researchers place on
funding amount versus duration is very similar to the estimates obtained using
our main specification shown above.

#### Heterogeneous preferences

As discussed previously, we expect the researchers who have the strongest
preference for money versus time (larger $\gamma$) to be those that discount
the future at higher rates, are capital-intensive, receive more direct utility
from grants, are less risk averse, and can more easily access research
funding.

Appendix C.4 contains the results from regressions where we split the sample
along these five dimensions, using either direct or proxy measures, and
estimate separate $\gamma$ parameters within sub-samples. In each case, we
find results that align with the predictions. Appendix C.4 also reports on
more tests of heterogeneity in $\gamma$ across other dimensions including
field of study and tenure status. Notably, the variation across these
dimensions is often smaller than what we observe across the five dimensions
motivated by our intuitive theory.

We can obtain some sense of the aggregate amount of heterogeneity in the
sample. To do so, we construct a summary metric that is the sum of these five
variables (after standardizing each variable), which we construct in a way
that larger values should correspond to a stronger preference for money over
time. We then split the sample into five equal groups using the quintiles of
this summary metric and estimate a $\gamma$ for each sub-sample.

The results of this exercise are shown in Table 8, which indicates that the
range of $\gamma$ in our sample spans (at least) from 0.7 to 0.8. This
corresponds to roughly a 40% difference in the marginal rate of substitution
across the full sample – there are some researchers who value money over time
nearly twice as much as others, and our stylized theory of scientific
production provides some guidance as to which researchers will have stronger
preferences for larger or longer grants.

Table 8: Heterogeneity in the amount-duration tradeoff

| (1) | (2) | (3) | (4) | (5)
---|---|---|---|---|---
$\gamma$ | 0.725∗∗∗ | 0.750∗∗∗ | 0.751∗∗∗ | 0.801∗∗∗ | 0.814∗∗∗
| (0.0146) | (0.0161) | (0.0170) | (0.0152) | (0.0142)
m.r.s. | 70,980 | 62,846 | 62,473 | 47,162 | 43,719
| [9.1%] | [8.0%] | [8.0%] | [6.0%] | [5.6%]
$N$ obs. | 835 | 835 | 835 | 835 | 835
Sub-sample | 1 | 2 | 3 | 4 | 5
| low |  $\leftarrow$ age $\rightarrow$ | high
| low |  $\leftarrow$ capital intensity $\rightarrow$ | high
| low |  $\leftarrow$ soft money share $\rightarrow$ | high
| low |  $\leftarrow$ risk tolerance $\rightarrow$ | high
| low |  $\leftarrow$ fundraising productivity $\rightarrow$ | high

> _Note_ : Larger $\gamma$ values reflect a stronger taste for grant size
> compared to duration; (m.r.s.) marginal rate of substitution ($ per year) is
> reported at the means and is shown as a percentage of average grant size in
> brackets; robust standard errors in parentheses; ∗ $p<0.1$, ∗∗ $p<0.05$, ∗∗∗
> $p<0.01$. The “sub-sample” row refers to the quintiles of the heterogeneity
> metric, which is the sum of the five standardized variables hypothesized to
> influence researchers’ preferences as illustrated at the bottom of the
> table.

###  5.4 Funders’ preferences

How well are funders’ preferences for grant designs aligned with researchers’?
Having funders undertake a thought experiment such as the one described above
is not feasible. However, we can get an initial sense of funders’ preferences
from the following two exercises: (1) a case study of a unique grant program
at the NIH that undertook the tradeoff of grant size and duration, and (2) an
analysis of the realized grant distribution (as shown in Figure 1).

#### Case study: The “Maximizing Investigators’ Research Award”

As noted in our introduction, in 2015 the National Institute of General
Medical Sciences at the NIH introduced the Maximizing Investigators’ Research
Awards (MIRA) program to provide grants that were longer than the traditional
grants that dominate their budget, but included limitations on the total
amount of funding researchers could receive. The program was described as
providing “award levels somewhat lower than current support in return for
increased award length, stability, flexibility, and reduced administrative
burden” compared to the traditional research project grant
mechanisms.313131See this FAQ by the NIH.

The complex way in which the program was implemented limits our ability to
conduct any formal analyses. But summary statistics reported by the NIH and
survey data from a reputable third party suggest that the program offered
researchers funding for approximately 30% longer and reduced their total
funding level over that period by 10–20%.323232The funding duration changes
are based on estimates reported by the NIH here. The funding level changes are
based on estimates reported by the Genetics Society of America here. These
magnitudes imply a $\gamma^{\text{MIRA}}$ of approximately 0.6–0.75, which is
about 15% smaller than the same parameter for researchers and implies the NIH
is more willing to trade off size for duration.

The MIRA program is still in its early stages, but preliminary feedback
indicated a mixed response with some researchers claiming they were “happy to
trade less funding for more stability of funding and more flexibility to
pursue new research directions,” while others noted that the reductions in the
grant size “might be too severe for labs to be able to continue doing their
science at the current level, and that people might have to be laid
off.”333333See this commentary by the Genetics Society of America for more.
These quotes are consistent with the relative difference between the
$\gamma^{\text{MIRA}}$ implied by the program design and the $\gamma$ we
estimate for our sample.

#### Analysis of realized grant design distribution

We can make some inferences about funders’ preferences more generally based on
the observed distribution of grant designs shown in Figure 1. Appendix D
details an exercise that treats all funders as a multi-product monopolist
facing inelastic demand from researchers. In this case, just as changes in a
monopolists prices reflects changes in marginal costs, the variation in the
number of grants of different designs reflects the “aggregate funder’s”
relative preference for awarding grants of different amount and duration.

This revealed preference approach yields an estimate of
$\gamma^{\text{funder}}\approx 0.4$. Intuitively, we should expect funders to
be rather willing to trade off grant amount for duration since there are much
tighter constraints on their annual budgets compared to the constraints on
their time horizons. More importantly, this exercise suggests that funders are
currently providing researchers with a much more “long-short” set of grant
opportunities than the researchers would prefer. Perhaps this is why most of
the effects of grant designs on researchers’ strategies seen in Section 4
(i.e., Table 3) are driven by variation in funding amounts as opposed to
duration. Whether the effects of grant design would be different in
alternative equilibria (e.g., in other populations such as academic
institutions in Europe, which have very different structures, or developing
countries, which have very different levels of inputs) is an interesting open
question.

##  6 Discussion

Understanding how grant design can be used to manage science requires
understanding the incentives and institutions that transform inputs and
outputs into objects that researchers value (e.g., job security, salary,
prestige). Practical examples of such factors include researchers’ taste for
science (Stern, 2004; Roach and Sauermann, 2010), the tenure process and
output measurement schemes (MacLeod and Urquiola, 2020), intellectual property
regimes (Hvide and Jones, 2018), the nature of competition within fields (Hill
and Stein, 2019), gate-keeping (Azoulay et al., 2019a ), and other social
factors more generally (Shapin, 1995). Our analyses provide new insights about
grant designs given the incentives and institutions surrounding researchers.
We investigate both the treatment and selection effects that can be induced by
grants with different designs.

Grants appear to affect researchers’ strategies in some intuitive and some
counter-intuitive ways. As predicted by prior theories, longer grants increase
researchers’ plans to take risks in their work, but only if the researcher has
the job security of tenure. This finding aligns with the work of Azoulay et
al., (2011), but provides new evidence as to some important boundary
conditions. Furthermore, these longer grants reduce _all_ researchers’ focus
on speed. More work remains to be done on the connection between these two
strategies and its implications for optimal grant design.343434See Nanda and
Rhodes-Kropf, (2017) for a discussion of how alternative innovation policies
can influence the types of projects funders are willing to support, and the
importance of understanding how this selection process influences the
extensive margin.

In contrast to conventional theory, we find that larger grants promote more
“exploit” strategies and smaller grants promote more “explore” strategies.
This may be partly driven by the subjective nature of our definitions of
strategy. But it may also signal some unique features of the scientific
production function that have not yet received much attention. We can
rationalize this result either with a model of uncertain production by risk-
averse researchers, or with a scarcity of inputs that are especially
productive when starting new projects. Further work to estimate the supply of,
demand for, and productivity of specific scientific inputs (e.g., graduate
student labor, data, equipment) is certainly an important avenue for future
work and is increasingly becoming possible using linked administrative data
(e.g., Lane et al., 2015; Chang et al., 2019).

When scaled to reflect the degree of grant design variation we see in
practice, these treatment effects are small. In the example of the NIH’s use
of MIRA grants versus traditional R01 grants (where the MIRA grant is roughly
15% smaller and 30% longer than the R01 grant), our estimates imply that
receiving one or the other would only change researchers’ probabilities of
choosing certain strategies by about 1 percentage point. But these (small)
treatment effects are not the full story.

Since the vast majority of grants require researchers to self-select their
grant pursuits, the usefulness of grant designs depends both on these
treatment effects as well as the presence of any selection effects — grant
designs may change the number and composition of researchers who pursue them.
To this point, our second thought experiment sheds new light on researchers’
preferences over grant design. Our results indicate that while there is some
willingness by researchers to tradeoff grant size (total grant $) for duration
(years grant $ are available), they value relative changes in size nearly four
times as much as changes in duration. The $\gamma$ parameter we estimate
subsumes several structural parameters, and our heterogeneity analyses
indicate there is some significant variation in preferences across
researchers. Disentangling this heterogeneity further to better understand
selection effects would require a much richer model and more complicated
experiments. For example, we use age as a proxy for researchers discount
rates, but it would be important to disentangle researchers’ actual discount
rates versus dynamic incentives they face (e.g., due to career concerns) over
their life-cycle. Overall, our results from this second thought experiment
indicate that the selection effects of grant design and how they might
influence the composition of researchers in an area are likely more important
than the treatment effect of how those grant designs might influence _how_
researchers conduct their science.

A key limitation of this paper is that we rely on non-incentivized stated
preferences from a small sample of researchers. As noted by Stantcheva,
(2022), this allows us to “create our own variation” and measure variables
that are difficult or impossible to observe in existing datasets (e.g.
bibliometric data). Nonetheless, this comes at the cost of having to assume
that these stated preferences are informative about actual behavior and that
our sample is representative of the population. Our investigations into
respondents’ attention and accuracy (using salary comparisons) and their
representativeness (using data comparisons and sample selection correction
techniques) suggest that these issues may not loom large in this specific
survey. Our results indicate that grants can be instrumental, and that the
large costs of field experiments with stronger guarantees of
representativeness may be worth the investment to better understand how grant
design can be used to manage science.

More generally, it is still clear that grants are a key tool for public
investments in science (Azoulay et al., 2019b ), can have important effects on
the research workforce (Tham et al., 2023), and, if large enough, can be used
to influence researchers’ scientific directions (Myers, 2020) even if they may
not have large effects on _how_ researchers pursue those new directions.
Recent evidence has highlighted the interaction of public and private funding
of science (Babina et al., 2023), which introduces more dimensions of funding
arrangements (e.g., intellectual property, control rights) that may play an
important role in influencing researchers’ decisions.

## References

  * Aghion et al., (2013) Aghion, P., Van Reenen, J., and Zingales, L. (2013).  Innovation and institutional ownership.  American Economic Review, 103(1):277–304.
  * Agrawal et al., (2016) Agrawal, A., Goldfarb, A., and Teodoridis, F. (2016).  Understanding the changing structure of scientific inquiry.  American Economic Journal: Applied Economics, 8(1):100–128.
  * Andrews and Kasy, (2019) Andrews, I. and Kasy, M. (2019).  Identification of and correction for publication bias.  American Economic Review, 109(8):2766–94.
  * Arrow, (1962) Arrow, K. (1962).  Economic welfare and the allocation of resources for invention.  In The Rate and Direction of Inventive Activity: Economic and Social Factors, pages 609–626. Princeton University Press.
  * (5) Azoulay, P., Fons-Rosen, C., and Zivin, J. S. G. (2019a).  Does science advance one funeral at a time?  American Economic Review, 109(8):2889–2920.
  * (6) Azoulay, P., Graff Zivin, J., Li, D., and Sampat, B. (2019b).  Public R&D investments and private-sector patenting: Evidence from NIH funding rules.  Review of Economic Studies, 86(1):117–152.
  * Azoulay et al., (2011) Azoulay, P., Graff Zivin, J. S., and Manso, G. (2011).  Incentives and creativity: Evidence from the academic life sciences.  RAND Journal of Economics, 42(3):527–554.
  * Azoulay and Li, (2021) Azoulay, P. and Li, D. (2021).  Scientific grant funding.  In Goolsbee, A. and Jones, B. F., editors, Innovation and Public Policy, chapter 4, pages 117–150. University of Chicago Press.
  * Babina et al., (2023) Babina, T., He, A. X., Howell, S. T., Perlman, E. R., and Staudt, J. (2023).  Cutting the innovation engine: How federal funding shocks affect university patenting, entrepreneurship, and publications.  Quarterly Journal of Economics, 138(2):895–954.
  * Blackwell and Olson, (2022) Blackwell, M. and Olson, M. P. (2022).  Reducing model misspecification and bias in the estimation of interactions.  Political Analysis, 30(4):495–514.
  * Boudreau et al., (2016) Boudreau, K. J., Guinan, E. C., Lakhani, K. R., and Riedl, C. (2016).  Looking across and looking beyond the knowledge frontier: Intellectual distance, novelty, and resource allocation in science.  Management Science, 62(10):2765–2783.
  * Brown et al., (2009) Brown, J. R., Fazzari, S. M., and Petersen, B. C. (2009).  Financing innovation and growth: Cash flow, external equity, and the 1990s R&D boom.  Journal of Finance, 64(1):151–185.
  * Bryan and Lemus, (2017) Bryan, K. and Lemus, J. (2017).  The direction of innovation.  Journal of Economic Theory, 172:247–272.
  * Bryan et al., (2022) Bryan, K. A., Lemus, J., and Marshall, G. (2022).  R&D competition and the direction of innovation.  International Journal of Industrial Organization, 82:102841.
  * Chang et al., (2019) Chang, W.-Y., Cheng, W., Lane, J., and Weinberg, B. (2019).  Federal funding of doctoral recipients: What can be learned from linked data.  Research Policy, 48(6):1487–1492.
  * (16) Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2018a).  Double/debiased machine learning for treatment and structural parameters: Double/debiased machine learning.  Econometrics Journal, 21(1):C1–C68.
  * (17) Chernozhukov, V., Demirer, M., Duflo, E., and Fernandez-Val, I. (2018b).  Generic machine learning inference on heterogeneous treatment effects in randomized experiments, with an application to immunization in india.  National Bureau of Economic Research Working Paper 24678.
  * Chernozhukov et al., (2015) Chernozhukov, V., Hansen, C., and Spindler, M. (2015).  Post-selection and post-regularization inference in linear models with many controls and instruments.  American Economic Review, 105(5):486–90.
  * Clancy, (2023) Clancy, M. (2023).  Biases against risky research.  New Things Under the Sun, retrieved from https://www.newthingsunderthesun.com/pub/41nqfjgh.
  * Cohen et al., (2020) Cohen, W. M., Sauermann, H., and Stephan, P. (2020).  Not in the job description: The commercial activities of academic scientists and engineers.  Management Science, 66(9):4108–4117.
  * Curty et al., (2017) Curty, R. G., Crowston, K., Specht, A., Grant, B. W., and Dalton, E. D. (2017).  Attitudes and norms affecting scientists’ data reuse.  PLoS ONE, 12(12):e0189288.
  * Dasgupta and David, (1994) Dasgupta, P. and David, P. A. (1994).  Toward a new economics of science.  Research Policy, 23(5):487–521.
  * De Vrieze, (2017) De Vrieze, J. (2017).  With this new system, scientists never have to write a grant application again.  Science Insider.
  * Digital Science, (2018) Digital Science (2018).  Dimensions [software].  Available from https://app.dimensions.ai. Accessed in 2023, under licence agreement.
  * Dixit and Pindyck, (1994) Dixit, A. K. and Pindyck, R. S. (1994).  Investment under uncertainty.  Princeton University Press.
  * Ederer and Manso, (2013) Ederer, F. and Manso, G. (2013).  Is pay for performance detrimental to innovation?  Management Science, 59(7):1496–1513.
  * Fox and Stephan, (2001) Fox, M. F. and Stephan, P. E. (2001).  Careers of young scientists: Preferences, prospects and realities by gender and field.  Social Studies of Science, 31(1):109–122.
  * Franzoni and Stephan, (2022) Franzoni, C. and Stephan, P. (2022).  Uncertainty and risk-taking in science: Meaning, measurement and management in peer review of research proposals.  Research Policy, page 104706.
  * Froot et al., (1993) Froot, K. A., Scharfstein, D. S., and Stein, J. C. (1993).  Risk management: Coordinating corporate investment and financing policies.  Journal of Finance, 48(5):1629–1658.
  * Furman et al., (2012) Furman, J. L., Murray, F., and Stern, S. (2012).  Growing stem cells: The impact of federal funding policy on the us scientific frontier.  Journal of Policy Analysis and Management, 31(3):661–705.
  * Furman and Stern, (2011) Furman, J. L. and Stern, S. (2011).  Climbing atop the shoulders of giants: The impact of institutions on cumulative research.  American Economic Review, 101(5):1933–1963.
  * Hagstrom, (1974) Hagstrom, W. O. (1974).  Competition in science.  American Sociological Review, pages 1–18.
  * Hall, (2002) Hall, B. H. (2002).  The financing of research and development.  Oxford review of economic policy, 18(1):35–51.
  * Heckman, (1979) Heckman, J. J. (1979).  Sample selection bias as a specification error.  Econometrica, 47(1):153–161.
  * Hellmann and Thiele, (2011) Hellmann, T. and Thiele, V. (2011).  Incentives and innovation: A multitasking approach.  American Economic Journal: Microeconomics, 3(1):78–128.
  * Hill and Stein, (2019) Hill, R. and Stein, C. (2019).  Scooped! Estimating rewards for priority in science.  Mimeo.
  * Hill and Stein, (2021) Hill, R. and Stein, C. (2021).  Race to the bottom: Competition and quality in science.  Mimeo.
  * Hoegl et al., (2008) Hoegl, M., Gibbert, M., and Mazursky, D. (2008).  Financial constraints in innovation projects: When is less more?  Research Policy, 37(8):1382–1391.
  * Holmstrom, (1989) Holmstrom, B. (1989).  Agency costs and innovation.  Journal of Economic Behavior & Organization, 12(3):305–327.
  * Hottenrott and Peters, (2012) Hottenrott, H. and Peters, B. (2012).  Innovative capability and financing constraints for innovation: more money, more innovation?  Review of Economics and Statistics, 94(4):1126–1142.
  * Hvide and Jones, (2018) Hvide, H. K. and Jones, B. F. (2018).  University innovation and the professor’s privilege.  American Economic Review, 108(7):1860–1898.
  * Imai and Ratkovic, (2013) Imai, K. and Ratkovic, M. (2013).  Estimating treatment effect heterogeneity in randomized program evaluation.  Annals of Applied Statistics, 7(1):443–470.
  * Johnston et al., (2015) Johnston, S. C., Desmond-Hellmann, S., Hauser, S., Vermillion, E., and Mia, N. (2015).  Predictors of negotiated NIH indirect rates at US institutions.  PLoS ONE, 10(3):e0121273.
  * Jones et al., (2019) Jones, D., Molitor, D., and Reif, J. (2019).  What do workplace wellness programs do? Evidence from the Illinois workplace wellness study.  Quarterly Journal of Economics, 134(4):1747–1791.
  * Kasy, (2021) Kasy, M. (2021).  Of forking paths and tied hands: Selective publication of findings, and what economists should do about it.  Journal of Economic Perspectives, 35(3):175–192.
  * Kerr and Nanda, (2015) Kerr, W. R. and Nanda, R. (2015).  Financing innovation.  Annual Review of Financial Economics, 7:445–462.
  * Krieger et al., (2022) Krieger, J., Li, D., and Papanikolaou, D. (2022).  Missing novelty in drug development.  Review of Financial Studies, 35(2):636–679.
  * Lane et al., (2015) Lane, J. I., Owen-Smith, J., Rosen, R. F., and Weinberg, B. A. (2015).  New linked data on research investments: Scientific workforce, productivity, and public value.  Research policy, 44(9):1659–1671.
  * Lerner and Wulf, (2007) Lerner, J. and Wulf, J. (2007).  Innovation and incentives: Evidence from corporate R&D.  Review of Economics and Statistics, 89(4):634–644.
  * Levecque et al., (2017) Levecque, K., Anseel, F., De Beuckelaer, A., Van der Heyden, J., and Gisle, L. (2017).  Work organization and mental health problems in PhD students.  Research Policy, 46(4):868–879.
  * Levin and Stephan, (1991) Levin, S. G. and Stephan, P. E. (1991).  Research productivity over the life cycle: Evidence for academic scientists.  American Economic Review, 81(1):114–132.
  * Loury, (1979) Loury, G. C. (1979).  Market structure and innovation.  Quarterly Journal of Economics, 93(3):395–410.
  * MacLeod and Urquiola, (2020) MacLeod, W. B. and Urquiola, M. (2020).  Why does the US have the best research universities? Incentives, resources, and virtuous circles.  National Bureau of Economic Research Working Paper 28279.
  * Madsen and Aagaard, (2020) Madsen, E. B. and Aagaard, K. (2020).  Concentration of danish research funding on individual researchers and research topics: Patterns and potential drivers.  Quantitative Science Studies, 1(3):1159–1181.
  * Mandler, (2017) Mandler, M. (2017).  The benefits of risky science.  Economic Journal, 127(603):1495–1526.
  * Manso, (2011) Manso, G. (2011).  Motivating innovation.  Journal of Finance, 66(5):1823–1860.
  * Merton, (1957) Merton, R. K. (1957).  Priorities in scientific discovery.  American Sociological Review, 22(6):635–659.
  * Merton, (1968) Merton, R. K. (1968).  The matthew effect in science: The reward and communication systems of science are considered.  Science, 159(3810):56–63.
  * Moreau and Dahl, (2005) Moreau, C. P. and Dahl, D. W. (2005).  Designing the solution: The impact of constraints on consumers’ creativity.  Journal of Consumer research, 32(1):13–22.
  * Murray et al., (2016) Murray, F., Aghion, P., Dewatripont, M., Kolev, J., and Stern, S. (2016).  Of mice and academics: Examining the effect of openness on innovation.  American Economic Journal: Economic Policy, 8(1):212–252.
  * Myers, (2020) Myers, K. (2020).  The elasticity of science.  American Economic Journal: Applied Economics, 12(4):103–134.
  * Myers et al., (2023) Myers, K. R., Tham, W. Y., Thursby, J. G., Thursby, M. C., Cohodes, N., Lakhani, K. R., Mural, R., and Xu, Y. (2023).  New facts and data about professors and their research.  Mimeo.
  * Myers et al., (2020) Myers, K. R., Tham, W. Y., Yin, Y., Cohodes, N., Thursby, J. G., Thursby, M. C., Schiffer, P., Walsh, J. T., Lakhani, K. R., and Wang, D. (2020).  Unequal effects of the COVID-19 pandemic on scientists.  Nature Human Behaviour, 4(9):880–883.
  * Nanda and Rhodes-Kropf, (2017) Nanda, R. and Rhodes-Kropf, M. (2017).  Innovation policies.  In Furman, J., Gawer, A., Silverman, B. S., and Stern, S., editors, Entrepreneurship, Innovation, and Platforms, volume 37, pages 37–80. Emerald Publishing Limited.
  * National Science Foundation, (2002) National Science Foundation (2002).  National Science Foundation report on efficiency of grant size and duration: Principal investigator FY 2001 grant award survey and institutional survey.
  * National Science Foundation, (2023) National Science Foundation (2023).  Higher Education Research and Development Survey (HERD).  Available from https://www.nsf.gov/statistics/srvyherd/. Accessed in 2023\.
  * Nelson, (1959) Nelson, R. (1959).  The simple economics of basic scientific research.  Journal of Political Economy, 67(3):297–306.
  * Pashler and Harris, (2012) Pashler, H. and Harris, C. R. (2012).  Is the replicability crisis overblown? Three arguments examined.  Perspectives on Psychological Science, 7(6):531–536.
  * Philipps, (2022) Philipps, A. (2022).  Research funding randomly allocated? A survey of scientists’ views on peer review and lottery.  Science and Public Policy, 49(3):365–377.
  * Roach and Sauermann, (2010) Roach, M. and Sauermann, H. (2010).  A taste for science? phd scientists’ academic orientation and self-selection into research careers in industry.  Research policy, 39(3):422–434.
  * Royal Society of New Zealand, (2004) Royal Society of New Zealand (2004).  Survey of 2001–2003 Marsden Fast Start Grant recipients.
  * Sauermann and Cohen, (2010) Sauermann, H. and Cohen, W. M. (2010).  What makes them tick? Employee motives and firm innovation.  Management Science, 56(12):2134–2153.
  * Sauermann and Roach, (2014) Sauermann, H. and Roach, M. (2014).  Not all scientists pay to be scientists: PhDs’ preferences for publishing in industrial employment.  Research Policy, 43(1):32–47.
  * Scopelliti et al., (2014) Scopelliti, I., Cillo, P., Busacca, B., and Mazursky, D. (2014).  How do financial constraints affect creativity?  Journal of Product Innovation Management, 31(5):880–893.
  * Semenova and Chernozhukov, (2021) Semenova, V. and Chernozhukov, V. (2021).  Debiased machine learning of conditional average treatment effects and other causal functions.  Econometrics Journal, 24(2):264–289.
  * Shapin, (1995) Shapin, S. (1995).  Here and everywhere: Sociology of scientific knowledge.  Annual Review of Sociology, 21(1):289–321.
  * Shortlidge and Eddy, (2018) Shortlidge, E. E. and Eddy, S. L. (2018).  The trade-off between graduate student research and teaching: A myth?  PLoS ONE, 13(6):e0199576.
  * Smaldino and McElreath, (2016) Smaldino, P. E. and McElreath, R. (2016).  The natural selection of bad science.  Royal Society Open Science, 3(9):160384.
  * Stantcheva, (2022) Stantcheva, S. (2022).  How to run surveys: A guide to creating your own identifying variation and revealing the invisible.  National Bureau of Economic Research Working Paper 30527.
  * Starr and MacMillan, (1990) Starr, J. A. and MacMillan, I. C. (1990).  Resource cooptation via social contracting: Resource acquisition strategies for new ventures.  Strategic Management Journal, pages 79–92.
  * Stephan, (1996) Stephan, P. E. (1996).  The economics of science.  Journal of Economic literature, 34(3):1199–1235.
  * Stern, (2004) Stern, S. (2004).  Do scientists pay to be scientists?  Management Science, 50(6):835–853.
  * Teodoridis, (2018) Teodoridis, F. (2018).  Understanding team knowledge production: The interrelated roles of technology and expertise.  Management Science, 64(8):3625–3648.
  * Tham, (2023) Tham, W. Y. (2023).  Science, interrupted: Funding delays reduce research activity but having more grants helps.  PLoS ONE, 18(4):e0280576.
  * Tham et al., (2023) Tham, W. Y., Staudt, J., Perlman, E. R., and Cheng, S. (2023).  The effect of funding delays on the research workforce: Evidence from tax records.  Mimeo.
  * Tian and Wang, (2014) Tian, X. and Wang, T. Y. (2014).  Tolerance for failure and corporate innovation.  Review of Financial Studies, 27(1):211–255.
  * Tibshirani et al., (2022) Tibshirani, J., Athey, S., Sverdrup, E., and Wager, S. (2022).  grf: Generalized Random Forests.  https://CRAN.R-project.org/package=grf, R package version 2.2.1 edition.
  * Veugelers et al., (2022) Veugelers, R., Wang, J., and Stephan, P. (2022).  Do funding agencies select and enable risky research: Evidence from ERC using novelty as a proxy of risk taking.  National Bureau of Economic Research Working Paper w30320.
  * Wager and Athey, (2018) Wager, S. and Athey, S. (2018).  Estimation and inference of heterogeneous treatment effects using random forests.  Journal of the American Statistical Association, 113(523):1228–1242.
  * Wang et al., (2018) Wang, J., Lee, Y.-N., and Walsh, J. P. (2018).  Funding model and creativity in science: Competitive versus block funding and status contingency effects.  Research Policy, 47(6):1070–1083.
  * Wang et al., (2017) Wang, J., Veugelers, R., and Stephan, P. (2017).  Bias against novelty in science: A cautionary tale for users of bibliometric indicators.  Research Policy, 46(8):1416–1436.
  * Westfall and Young, (1993) Westfall, P. H. and Young, S. S. (1993).  Resampling-based multiple testing: Examples and methods for p-value adjustment, volume 279.  John Wiley & Sons.
  * Wu et al., (2019) Wu, L., Wang, D., and Evans, J. A. (2019).  Large teams develop and small teams disrupt science and technology.  Nature, 566(7744):378–382.

.

##  Appendix A Additional summary statistics

###  A.1 Full summary statistics for covariates

Tables A1-A2 report the mean and standard deviation for all variables ever
used in the analyses. If the count of observations is less than the full
sample (4,175), it is because respondents were not required to answer the
question and some chose not to — this only is relevant for the socio-
demographic questions. Figure A1 plots the distribution of expected research
funding across the five broad fields to illustrate the pervasiveness of demand
for research grants.

Table A1: Full summary statistics for covariates
| count | mean | sd
---|---|---|---
Professional |  |  |
_Rank_ {0,1} |  |  |
Adjunct, clinical, teaching, other | 4,175 | 0.06 | 0.23
Assistant | 4,175 | 0.25 | 0.44
Associate | 4,175 | 0.25 | 0.43
Emeritus or retired | 4,175 | 0.03 | 0.17
Full | 4,175 | 0.41 | 0.49
_Tenure_ {0,1} |  |  |
Not on tenure track, not applicable | 4,175 | 0.20 | 0.40
On tenure track, pre-tenure | 4,175 | 0.22 | 0.41
Tenured | 4,175 | 0.59 | 0.49
_Salary_ |  |  |
Base | 4,175 | 120,649.10 | 79,692.66
Grant-sponsored or | 4,175 | 25,336.53 | 45,407.18
Supplementary | 4,175 | 5,142.51 | 16,566.07
Clinical | 4,175 | 5,692.22 | 37,375.73
Outside activities | 4,175 | 7,798.80 | 29,703.94
Guaranteed research funding | 4,175 | 430,699.40 | 1,085,486.21
Research fundraising expectations | 4,175 | 549,238.32 | 1,055,926.36
Total work hrs./week | 4,175 | 49.88 | 13.06
_Share of work time_ [0,1] |  |  |
Research | 4,175 | 0.39 | 0.20
Fundraising | 4,175 | 0.09 | 0.11
Teaching | 4,175 | 0.27 | 0.17
Administration | 4,175 | 0.15 | 0.14
Clinical | 4,175 | 0.04 | 0.14
Other | 4,175 | 0.06 | 0.09
_Aggregate field_ [0,1] |  |  |
Humanities | 4,175 | 0.17 | 0.37
Medical & health | 4,175 | 0.19 | 0.39
Natural | 4,175 | 0.28 | 0.45
Social & mathematical | 4,175 | 0.16 | 0.36
Table A2: Full summary statistics for covariates, continued
| count | mean | sd
---|---|---|---
Socio-demographic |  |  |
Age | 4,084 | 49.31 | 12.52
Female {0,1} | 3,992 | 0.41 | 0.49
Married or in partnership {0,1} | 4,044 | 0.84 | 0.36
U.S. immigrant {0,1} | 4,088 | 0.27 | 0.44
3rd generation in U.S. or less {0,1} | 2,848 | 0.44 | 0.50
Household annual income | 4,045 | 260,819.53 | 215,128.41
Num. household dependents | 4,104 | 0.98 | 1.13
_Race/ethnicity_ {0,1} |  |  |
Asian | 4,175 | 0.12 | 0.33
Black | 4,175 | 0.03 | 0.17
Hispanic | 4,175 | 0.06 | 0.23
White | 4,175 | 0.78 | 0.42
Other | 4,175 | 0.05 | 0.22
Risk-taking |  |  |
Research, own belief | 4,175 | 4.63 | 2.35
Research, others’ belief | 4,175 | 4.56 | 2.40
Personal life | 4,175 | 5.25 | 2.12
Research |  |  |
Objective: ask or answer questions [0,10] | 4,175 | 4.85 | 2.57
_Output_ {0,1,2} |  |  |
Journal articles | 4,175 | 1.87 | 0.37
Books | 4,175 | 0.52 | 0.69
Materials, methods, or tools | 4,175 | 0.70 | 0.68
Products | 4,175 | 0.47 | 0.64
_Audience_ {0,1,2} |  |  |
Academics | 4,175 | 1.88 | 0.36
Policymakers | 4,175 | 0.84 | 0.69
Businesses and orgs. | 4,175 | 0.54 | 0.62
General public | 4,175 | 0.83 | 0.62
Other |  |  |
Inverse Mills Ratio | 4,175 | 2.40 | 0.10
Figure A1: Expected research funding by broad field

(a)

(b)

(c)

(d)

(e)

> _Note_ : Shows the within-field distributions of researchers’ total expected
> research funding over the next five years, which is the sum of existing
> funding and expected fundraising amounts over the same period.

###  A.2 Population and sample comparisons

Here we replicate the two tests of representativeness for the survey as also
done in Myers et al., (2023). First, A2 shows this comparison based on data
from the National Science Foundation’s HERD survey (National Science
Foundation, 2023), which reports institutional-level data on the total amount
of funding flows into all of the institutions in our population (recall, our
population was constructed using the HERD, which is why this data is available
for the full population).

Figure A2 reports the distributions and regression tests for mean differences
in six metrics of institution-level research funding that compare respondents
to the full set of researchers invited to participate (“Sample e-mailed”). The
distributions overlap to a large degree (see Panel a). We do estimate
statistically significant differences in means (see Panel b), but the
magnitudes of these differences are all in the range of approximately 4–6%.

Figure A2: Recruitment versus completion sample comparison per HERD metrics

(a)

| All sources, by type |  | All types, by source
---|---|---|---
| All | Federal | Non–fed. |  | Basic | Applied | Develop.
| (1) | (2) | (3) |  | (4) | (5) | (6)
Completed survey |  –35.01∗∗∗ |  –21.30∗∗∗ |  –13.71∗∗∗ |  |  –24.31∗∗∗ |  –7.170∗∗∗ |  –3.812∗∗∗
| (6.953) | (4.511) | (3.062) |  | (4.867) | (2.492) | (1.328)
Constant |  649.1∗∗∗ |  351.7∗∗∗ |  297.4∗∗∗ |  |  413.2∗∗∗ |  176.7∗∗∗ |  60.46∗∗∗
| (1.308) | (0.863) | (0.578) |  | (0.917) | (0.459) | (0.254)
% diff. | –5.4% | –6.1% | –4.6% |  | –5.9% | –4.1% | –6.3%
$N$ obs. | 130,785 | 130,785 | 130,785 |  | 130,785 | 130,735 | 128,169
(b)

> _Note_ : Panel (a) compares the distributions of e-mailed professors and
> respondents per the rank of their institution along each dimension of R&D
> funding. Panel (b) reports estimates from a regression of each sampled
> professor’s institutional R&D funding (in 2019 $-M) on a dummy for whether
> the sampled individual completed the survey; the “% diff.” row reports the
> mean difference in the measure as a percentage of the non-respondent average
> (i.e., it is the ratio of the two coefficients); robust standard errors in
> parentheses; ∗ $p<0.1$, ∗∗ $p<0.05$, ∗∗∗ $p<0.01$. All institutional data is
> from the 2019 NSF HERD (National Science Foundation, 2023).

Figure A3 reports the results of a similar exercise, instead using individual-
level data on researchers’ publication output and grant receipts. This data
was obtained by performing a fuzzy match of our population (i.e., using names
and institution data) to the Dimensions database (Digital Science, 2018),
which includes disambiguated researcher-level records. We focus on
researchers’ publications and grants during the twenty years prior (2003-2022)
and see very little differences between our respondents and the full set of
individuals invited to the survey. The distributions have strong overlap over
the full support (see Panel a), and the mean differences are all insignificant
and/or smaller than 7.5%.

Figure A3: Recruitment versus completion sample comparison per Dimensions
metrics

(a)

| Pub., | Pub. cites, | Grant, | Grant,
---|---|---|---|---
| count | normalized | count | total $
| (1) | (2) | (3) | (4)
Completed survey | –0.111 | –0.575 |  0.00878∗∗ | –5568.8
| (0.0688) | (0.807) | (0.00343) | (8519.4)
Constant |  2.906∗∗∗ |  20.59∗∗∗ |  0.117∗∗∗ |  121272.4∗∗∗
| (0.0158) | (0.185) | (0.000715) | (2093.0)
% diff. | –3.8% | –2.8% | 7.5% | –4.6%
$N$ obs. | 87,000 | 87,000 | 87,000 | 87,000
(b)

> _Note_ : Panel (a) compares the distributions of e-mailed professors and
> respondents per each dimension of individual-level publication output and
> grant funding per year (2003–2022). Panel (b) reports estimates from a
> regression of each sampled professor’s publication or grant metric on a
> dummy for whether the sampled individual completed the survey; the “% diff.”
> row reports the mean difference in the measure as a percentage of the non-
> respondent average (i.e., it is the ratio of the two coefficients); robust
> standard errors in parentheses; ∗ $p<0.1$, ∗∗ $p<0.05$, ∗∗∗ $p<0.01$. All
> publication and grant data is from the Dimensions database (Digital Science,
> 2018).

###  A.3 Randomized incentives

In order to implement the sample selection correction methods of Heckman,
(1979), we must have a source of variation in survey participation that is
uncorrelated with any of the focal parameters. With such variation in hand, a
control function can be constructed and included in the focal regressions to
condition on any variation in respondents’ answers that otherwise would have
been unobservable, endogenous, and led to biased estimates. Furthermore, the
statistical significance of this control function can shed light on the degree
to which selection into the survey on (relevant endogenous) unobservables may
have occurred.

To generate the necessary variation in participation, we implemented
randomized incentives and reminders in our e-mail strategy. Each e-mail was
randomly assigned to (1) one of four incentive arms comprising a gift card
lottery or a donation on the respondents’ behalf to a charity of their choice,
and (2) one of three reminder schedules including zero, one, or two reminders.
Figure 4(a) shows the raw data and statistical tests of the effects of these
arms (aggregating the incentive arms in Panel a, and assuming linear
separability of the treatments in the regressions shown in Panel b). There are
clear effects, with the “most treated” invites (i.e., both incentives and two
reminders) increasing the probability of survey participation by roughly 2.7
percentage points to roughly 4.5%, yielding an average response rate roughly
more than twice as large as the “least treated” invites (i.e., no reminders
and no incentives) whose response rate was 1.9%.

Figure A4: Completion rate by incentive and reminder arms

(a)

Either incentive |  0.00297∗∗
---|---
| (0.00117)
Both incentives |  0.00797∗∗∗
| (0.00141)
1 reminder |  0.0112∗∗∗
| (0.00114)
2 reminders |  0.0188∗∗∗
| (0.00119)
Constant |  0.0197∗∗∗
| (0.00107)
$N$ obs. | 131,672
(b)

> _Note_ : Panel (a) shows the average completion rate by each combination of
> incentive and reminder arm; Panel (b) shows estimates from a linear
> probability model of completion as a function of incentives and reminders;
> robust standard errors in parentheses; ∗ $p<0.1$, ∗∗ $p<0.05$, ∗∗∗ $p<0.01$.

##  Appendix B Additional results on strategic responses to grants

###  B.1 Main results on strategic responses, reporting sample selection
correction estimates

Table B1 replicates the results from the same analyses as reported in Table 3,
but here also reporting the coefficient estimates associated with the Inverse
Mills Ratio (IMR) covariate and the constant term. In all cases, we do not
reject the null hypothesis of no sample selection effects with none of the IMR
coefficients being significantly different from zero (Heckman, 1979). The
strong effects of the participation incentives and reminders reported in
Section A.3 suggests that the IMR control is valid in the sense that there is
non-trivial variation in the IMR driven by those randomized incentives and
reminders. We still cannot rule out that there are no other unobservable
dimensions of heterogeneity that are both correlated with researchers’
responsiveness to either the incentives or the reminders as well as our focal
parameters (i.e., researchers’ strategic responsiveness to grant designs).
However, we have no qualitative, theoretical, or other empirical reason to
suggest any specific sources of such lingering selection that we could
investigate further.

Table B1: Effect of grant designs on researchers’ strategies, reporting sample
selection coefficient

|  |  |  |  |  |  | Larger |  |
---|---|---|---|---|---|---|---|---|---
|  |  | More |  | New |  | ongoing |  | More
| Faster |  | risk |  | directions |  | projects |  | accurate
| (1) |  | (2) |  | (3) |  | (4) |  | (5)
log(Duration) | –0.056∗∗∗ |  | 0.012 |  | 0.008 |  | 0.018 |  | 0.018
| (0.015) |  | (0.016) |  | (0.015) |  | (0.015) |  | (0.011)
log(Amount) | –0.025∗∗∗ |  | 0.017∗∗ |  | –0.016∗∗ |  | 0.026∗∗∗ |  | –0.003
| (0.007) |  | (0.007) |  | (0.007) |  | (0.007) |  | (0.006)
IMR | –0.008 |  | 0.003 |  | 0.010 |  | –0.005 |  | 0.000
| (0.007) |  | (0.008) |  | (0.007) |  | (0.008) |  | (0.006)
Constant | 0.776∗∗∗ |  | 0.285∗∗∗ |  | 0.533∗∗∗ |  | 0.228∗∗ |  | 0.178∗∗
| (0.097) |  | (0.101) |  | (0.095) |  | (0.099) |  | (0.077)
dep. var. mean | 0.36 |  | 0.53 |  | 0.33 |  | 0.61 |  | 0.17
$N$ obs. | 4,175 |  | 4,175 |  | 4,175 |  | 4,175 |  | 4,175

> _Note_ : Shows results from OLS regressions where the dependent variables
> are indicators that equal one if the strategy listed at the top of the
> column was chosen as one of the “two most important changes” that
> researchers would make in response to receiving a grant of a given
> (randomized) size and duration. The IMR row reports the coefficient on the
> standardized Inverse Mills Ratio. Robust standard errors shown in
> parentheses; ∗ $p<0.1$, ∗∗ $p<0.05$, ∗∗∗ $p<0.01$.

###  B.2 Joint estimation of strategic responses

In the main text, we estimate separate five separate OLS models where we
regress an indicator variable for whether the researcher chose each strategy
on the (randomized) grant attributes they were provided with. This approach
abstracts away from the fact that the researcher is effectively making a
single, discrete choice decision about which pair of strategies to
select.353535Recall, the researcher is told they have received some grant and
are asked to choose the “top 2” most important changes to their research the
grant would enable. To more closely model the structure of this problem, we
can convert our data into researcher ($i$) by strategy ($j$) level
observations and estimate a logit model of the form:

$\Pr(\text{Strategy
change}_{ij}=1\,|\,D_{i},A_{i},X_{i})=\frac{\exp(\log(D_{i})\beta^{D}_{j}+\log(A_{i})\beta^{A}_{j}+\mathbf{X}\bm{\delta})}{1+\exp(\log(D_{i})\beta^{D}_{j}+\log(A_{i})\beta^{A}_{j}+\mathbf{X}\bm{\delta})}\,\,,$
(B1)

which allows the effect of the grant attributes to depend on the strategy
being considered. Now, instead of five OLS regressions, we have a single logit
regression. Table B2 reports the results of estimating Equation B1 either
without any covariates (Col. 1) or by using cross-fit partialing-out Lasso to
select and include some covariates (Col. 2) per Chernozhukov et al., 2018a .
In both cases, the results mirror those of our simpler, OLS-based approach
shown in the main text.

Table B2: Logit model of strategy changes in response to grant design

| (1) | (2)
---|---|---
More risk |  |
$\times$ log(Duration) | 0.048 | 0.050
| (0.062) | (0.063)
$\times$ log(Amount) | 0.069∗∗ | 0.069∗∗
| (0.030) | (0.030)
Faster |  |
$\times$ log(Duration) | –0.239∗∗∗ | –0.243∗∗∗
| (0.065) | (0.065)
$\times$ log(Amount) | –0.107∗∗∗ | –0.104∗∗∗
| (0.031) | (0.031)
New direction |  |
$\times$ log(Duration) | 0.034 | 0.043
| (0.066) | (0.067)
$\times$ log(Amount) | –0.073∗∗ | –0.079∗∗
| (0.031) | (0.032)
Ongoing bigger |  |
$\times$ log(Duration) | 0.078 | 0.083
| (0.063) | (0.064)
$\times$ log(Amount) | 0.111∗∗∗ | 0.117∗∗∗
| (0.030) | (0.031)
More accurate |  |
$\times$ log(Duration) | 0.125 | 0.122
| (0.082) | (0.083)
$\times$ log(Amount) | –0.019 | –0.019
| (0.040) | (0.040)
$L(\mathbf{X})$ |  | Y
$Ni$ obs. | 4,175 | 4,175
$Nij$ obs. | 20,875 | 20,875

> _Note_ : Reports the estimates of $\beta^{D}_{j}$ and $\beta^{S}_{j}$ for
> each of the strategic options (indexed by $j$) per Eq. B1. Col. 1 includes
> no covariates and Col. 2 includes any Lasso selected covariates. Robust
> standard errors shown in parentheses; ∗ $p<0.1$, ∗∗ $p<0.05$, ∗∗∗ $p<0.01$.

###  B.3 Best Linear Predictors of heterogeneity

Figure 2 in the main text indicates some notable heterogeneity in researchers’
responsiveness to grant designs. Those plots of the individual-level
Conditional Average Partial Effects (CAPEs) are all based on the high-
dimensional vector of covariates included in the causal forests (see Tables
A1-A2 for the full list of those covariates).

To better understand the sources of that heterogeneity in a more low-
dimensional way, Figure B1 reports some of the Best Linear Predictors of the
CAPEs (Semenova and Chernozhukov, 2021). Loosely speaking, the Best Linear
Predictor approach regresses an individual’s CAPE on some covariate in order
to test how the CAPE covaries with that observable, effectively testing
whether that covariate mediates the focal causal effect. To do this, we make
use of the best_linear_projection function within the grf package (Tibshirani
et al., (2022)).

There is no systematic way for choosing which observables to test here, so
Figure B1 reports only any observable which yields a $p$-value in a univariate
regression of less than 0.01. The “Univariate coefficient” reported is in
standard deviation units and relative to the mean CAPE (i.e., the mean
treatment effect). Thus, a univariate coefficient of 0.1 for Humanities would
indicate that the effect of the grant attribute on the focal strategy is 0.1
larger for researchers in that field compared to the rest of the sample.

Figure B1: Univariate Best Linear Predictors of heterogeneous treatment
effects

| Univar.
---|---
| coef.
_Size_ |
Outputs: Books | –0.021∗∗∗
_Duration_ |
_n/a_ |
(a)

| Univar.
---|---
| coef.
_Size_ |
_n/a_ |
_Duration_ |
Tenured | 0.087∗∗∗
Not on tenure-track | –0.101∗∗∗
(b)

| Univar.
---|---
| coef.
_Size_ |
Outputs: Books | 0.021∗∗∗
_Duration_ |
_n/a_ |
(c)

| Univar.
---|---
| coef.
_Size_ |
_n/a_ |
_Duration_ |
Outputs: Products | 0.039∗∗∗
(d)

| Univar.
---|---
| coef.
_Size_ |
_n/a_ |
_Duration_ |
_n/a_ |
(e)

> _Note_ : Reports the results from univariate Best Linear Predictors (BLP)
> regressions of the Conditional Average Partial Effects (CAPEs) on aggregate
> field indicators as well as other select covariates; only variables with
> univariate coefficient $p$-values of 0.01 or less are shown; “n/a” indicates
> that no covariates had $p$-values less than 0.01 in the BLP regressions. The
> mean effects are reported in Table 3. Robust standard errors, which account
> for the fact that the CAPEs are estimated objects, shown in parentheses; ∗∗∗
> $p<0.01$

###  B.4 Alternative estimation of heterogeneity

In the main analyses, we estimate researchers’ heterogeneous strategic
responses to grant design using the causal forest method of Wager and Athey,
(2018). That approach is appealing because it allows for the estimation of
unit-specific partial effects. However, in order to obtain a lower-dimensional
understanding of what covariates are influencing the heterogeneity captured by
the random forest, we must ex-post select variables of interest.

An alternative approach is to directly focus on identifying treatment-
covariate interaction terms that are “important” with the use of machine
learning tools such as the lasso. Here, we follow an algorithm inspired by the
approaches of Imai and Ratkovic, (2013) and Blackwell and Olson, (2022) that
uses two stages of lassos. The algorithm is as follows:

  1. 1.

Generate interaction terms between each randomized grant attributes (amount
$A$ and duration $D$) and all covariates $\mathbf{X}$

  2. 2.

Perform a lasso regression of a given dependent variable on
($A,D,A\times\mathbf{X},D\times\mathbf{X},\mathbf{X}$); let
$L^{1}_{A,D,\mathbf{X}}$ denote the subset of attribute-covariate interaction
terms selected in this step, i.e., from the possible set
$(A\times\mathbf{X},D\times\mathbf{X})$

  3. 3.

Perform a cross-fit partialing-out lasso linear regression of a given
dependent variable on ($A,D,L^{1}_{A,D,\mathbf{X}}$) using $\mathbf{X}$ as the
set of high-dimensional controls (Chernozhukov et al., 2015)

This approach relies on a separate penalty parameter to be used in the
selection of interaction terms (Step 2.) and in the selection of covariates
(Step 3.) as suggested by Imai and Ratkovic, (2013). And by using a post-
selection inference method (Chernozhukov et al., 2015), we are attempting to
avoid regularization biases as noted by Blackwell and Olson, (2022).

Table B3 reports the results from this alternative approach. Overall, the
results are consistent with the random forests approach. Notably, tenure
status is again observed to be a clear moderator of the effect of grant
duration on risk-taking. The main effect estimates also align with our other
estimation approach.

Table B3: Heterogeneous strategic responses to grant attributes

|  |  |  | Larger |
---|---|---|---|---|---
|  | More | New | ongoing | More
| Faster | risk | directions | projects | accurate
log(Duration) | –0.030 | –0.052∗∗ | 0.015 | 0.019 | 0.019∗
| (0.023) | (0.023) | (0.017) | (0.015) | (0.011)
$\times$ Male | –0.044 |  |  |  |
| (0.030) |  |  |  |
$\times$ Tenured |  | 0.094∗∗∗ |  |  |
|  | (0.030) |  |  |
$\times$ On t.t., not tenured |  |  | –0.045 |  |
|  |  | (0.035) |  |
$\times$ Audience: Policymakers |  |  |  |  | 0.015
|  |  |  |  | (0.011)
log(Amount) | –0.018 | 0.012 | –0.016∗∗ | 0.026∗∗∗ | –0.003
| (0.013) | (0.010) | (0.007) | (0.008) | (0.006)
$\times$ Age | 0.002 |  |  |  |
| (0.008) |  |  |  |
$\times$ Assistant professor | 0.027 |  |  |  |
| (0.028) |  |  |  |
$\times$ Full professor | –0.014 | 0.011 |  |  |
| (0.017) | (0.016) |  |  |
$\times$ On t.t., not tenured | –0.015 |  |  |  |
| (0.029) |  |  |  |
$\times$ Engineering & related | –0.036∗∗ |  |  |  |
| (0.018) |  |  |  |
$\times$ Total salary |  | 0.001 |  |  |
|  | (0.008) |  |  |
$\times$ Fundraising expect. |  | 0.005 |  |  |
|  | (0.006) |  |  |
$\times$ Research risk, own belief |  |  | –0.012 |  |
|  |  | (0.007) |  |
$\times$ Humanities & related |  |  |  | 0.009 |
|  |  |  | (0.018) |
Incl. main effect | Y | Y | Y | Y | Y
Lasso method | plug–in | plug–in | plug–in | plug–in | plug–in
$NX$ var. sel. | 11/62 | 19/62 | 7/62 | 4/62 | 7/62
dep. var. mean | 0.36 | 0.53 | 0.33 | 0.61 | 0.17
$N$ obs. | 4,175 | 4,175 | 4,175 | 4,175 | 4,175

> _Note_ : Robust standard errors shown in parentheses; ∗ $p<0.1$, ∗∗
> $p<0.05$, ∗∗∗ $p<0.01$.

###  B.5 Effects at the limits of grant design

To get a sense of the limits grant designs, Figure B2 plots the predicted
choice probabilities for each strategy given designs at the limits of the
dollar amount and duration explored in the thought experiment. These predicted
probabilities are based on the point estimates from the logistic regression
model. Thus, they are “generous” in that we do not force any point estimates
that do not reject the null (per their standard errors) to zero.

We find that a “MIRA-sized change in grant design” (i.e., an additional 30% in
duration and a 15% decline in amount) leads to no more than a 1 p.p. change in
the probability of any strategy being chosen. Using the full support of grant
designs from the thought experiment, the point estimates and confidence
intervals indicate that it takes a tremendous amount of differences in grant
design (i.e., plus or minus approximately $2 million and/or 8 years) to change
choice probabilities by an amount that could reasonably be considered
meaningful (i.e, approximately 10 p.p.). For comparison, $2 million is an
order of magnitude larger than the median funding expectations of professors
in the sample, and 8 years is roughly 3 times the average duration of non-
tenure-track professors contracts (approximately 2.5 years) and the average
amount of time before a pre-tenure professor’s tenure evaluation
(approximately 2.5 years). Table B4 summarizes these ranges based on estimates
from the full sample, as well as estimates from just the non-tenured and
tenured subsamples.

Figure B2: Choice probabilities at observed and limits of grant designs

> _Note_ : Reports the predicted probabilities of choosing a strategy when
> receiving (1) an average grant in the thought experiment (“Base”) , (2) a
> grant that is 30% longer and 15% smaller than average, reflecting the same
> relative change as the MIRA grants compared to traditional R01 grants at the
> NIH (“MIRA”); (3-6) grants with the largest or smallest dollar amount in the
> thought experiment
> ($A_{\text{max}}=\text{\$2,000,000};A_{\text{min}}=\text{\$100,000}$) and
> the longest or shortest duration in the thought experiment
> ($D_{\text{max}}=\text{2 years};D_{\text{min}}=\text{10 years}$). All
> predicted probabilities are based on the logit models.

Table B4: Span of probabilities at observed and limits of grant designs

|  |  |  |  | Point Estimate |  | 95% C.I.
---|---|---|---|---|---|---|---
| mean |  | MIRA |  | min | max |  | min | max
Panel (a): Full sample |  |  |  |  |  |  |  |  |
Faster | 0.34 |  | 0.33 |  | 0.29 | 0.46 |  | 0.27 | 0.50
More risk | 0.54 |  | 0.54 |  | 0.49 | 0.56 |  | 0.45 | 0.59
New directions | 0.33 |  | 0.33 |  | 0.30 | 0.36 |  | 0.27 | 0.40
Larger ongoing projects | 0.62 |  | 0.62 |  | 0.54 | 0.65 |  | 0.50 | 0.68
More accurate | 0.17 |  | 0.18 |  | 0.15 | 0.19 |  | 0.13 | 0.22
Panel (b): Non-tenured |  |  |  |  |  |  |  |  |
Faster | 0.40 |  | 0.39 |  | 0.36 | 0.48 |  | 0.32 | 0.54