Are there returns to risk taking in science, philanthropy, or public policy?

If you’re hoping to do good in the world, it makes sense to ask where your efforts will make the biggest impact. Some have claimed that high risk, high return projects are most promising because those areas are less crowded. For example, here’s a quote from Robert Reich’s essay on the role of philanthropic foundations in society:

When it comes to the ongoing work of experimentation, foundations have a structural advantage over market and state institutions: a longer time horizon. Once more, the lack of accountability may be a surprising advantage. . . foundations are not subject to earnings reports, impatient investors or stockholders, or short-term election cycles. Foundations, answerable only to the diverse preferences and ideas of their donors, with a protected endowment permitted to exist in perpetuity, may be uniquely situated to engage in the sort of high-risk, long-run policy innovation and experimentation that is healthy in a democratic society.

The Open Philanthropy Project outlines a similar approach in a post about their giving philosophy:

One of our core values is our tolerance for philanthropic “risk.” Our overarching goal is to do as much good as we can, and as part of that, we’re open to supporting work that has a high risk of failing to accomplish its goals. We’re even open to supporting work that is more than 90% likely to fail, as long as the overall expected value is high enough.

It seems intuitive that there are returns to risk taking but I was wondering if there were any datasets out there that would support this idea. Below I attempt to answer this question by looking at evidence from science, philanthropy, and public policy. I just finished a meta-analysis that combines the interventions from below that have similar units, so take a look at that as well.

Definitions

Before I continue, I think it makes sense to define the terms risk and return. By return, I mean the impact of an intervention using units like disability adjusted life years per dollar, benefit to cost ratios, or research citation counts. While some of these estimates are more complicated to construct than others, they all require making judgements about things like the value of a human life, the amount of suffering caused by different conditions, or the benefits from a highly cited paper.

The definition of the term risk is tricky to pin down. To some, it’s just a measure of the noisiness of an estimate and is measured using something like the standard deviation. To others, an intervention is only risky when it could potentially underperform some target or cause harm (e.g. downside risk). People seem to use the uncertainty and downside risk terms interchangeably, and I think both are useful, so I include both in my analysis where possible. When I use the term risk below, I’m talking about the standard deviation of the outcome and downside risk refers to the underperformance of the mean.

Here are how the values are calculated:

• risk = standard deviation = np.std(series)
• downside risk (semideviation) = np.sqrt((np.minimum(0.0, series - t)**2).sum()/series.size), where t is the mean intervention outcome

Finally, how do you determine if there is a return to risk taking? One approach would be to run a linear regression through the data and see if it has a positive slope. This is what I do for most of the sources below, but there’s a problem with this approach. To see why, imagine calculating the cost effectiveness of every possible action, including bogus things like “lighting $1000 on fire”. You’d end up with a lot of useless interventions that would mess up the slope of the linear regression. So another approach would be to just see if the frontier that encloses the top end of the estimates has a positive slope. In Modern Portfolio Theory, this frontier is called the efficient frontier:

I’ve written about this idea before but didn’t have the data to test the concept out until now. I stick with linear regression for this notebook, but I run the data through a portfolio optimization algorithm to create an efficient frontier in my meta-analysis blogpost.

Data Sources

It’s pretty difficult to find datasets that quantify their uncertainty while also using a cross-intervention measure of impact, so it’s taken me awhile to stumble across enough data to complete this analysis. The scrape.py file included in the repo for this projects outlines how I accessed and cleaned data from each source.

All the code and data for this post are available here.

Evidence from Public Policy

First, I look at a dataset from the Washington State Institute for Public Policy (WSIPP). The WSIPP evaluates evidence based public policies and completes detailed benefit-cost analyses using monte carlo methods. The end result is a list of benefit-cost ratios along with metrics like the chance that the benefit-cost ratio is positive.

The measure of risk I’m using here (the chance costs exceed benefits) sets a really low bar. This ignores the upside of an intervention and much of the downside until the benefit cost ratio is below one. It also counts a project with a very low downside the same of one with only a marginally low downside because they’re just counting up benefit-cost ratios < 1 and dividing by the total number of monte carlo runs. The benefit of this metric is that it is easy to interpret, but I wish they would include a standard deviation as well.

Below is a plot of the intervention rank and the benefit-cost ratio. It’s clear that some interventions outperform others by a few orders of magnitude. Another interesting finding is that the distribution might be two tailed, with some outlying performers on the bad end as well.

Next, I plot the chance costs exceed benefits (an imperfect proxy for downside risk) against the benefit-cost ratio. I derive the chance costs exceed benefits from WSIPP’s chance benefits exceed costs value, which they calculate by counting results from their monte carlo simulations. This measure doesn’t take into account the scale of good/poor performance, but it’s the best we can get without access to their models.

The end result is that there doesn’t seem to be much of a return to this measure of risk.

Evidence from Public Health

The next dataset I look it is from the Disease Control Priorities Project (DCP2), which comes up with comprehensive estimates of the cost effectiveness of different treatments in developing countries. The original source is a table in the DCP2 report, which Jeff Kaufmann made into a CSV. I selected the interventions with $/DALY units, eliminated any with zero or near zero spread (because they likely came from the same estimate), and only selected the estimates from sub-saharan Africa. Finally I converted $/DALY units to DALY/1000USD so a bigger number has a higher impact.

Using the spread isn’t very rigorous and might bias the results towards understudied areas with few estimates (e.g. an intervention with only a single estimate has spread of 0), but it’s the only measure of uncertainty available here.

These estimates follow a similar pattern to the WSIPP data, with the top interventions a few orders of magnitude better than the worst.

So it seems there might be returns to risk taking when using the spread as the (somewhat imperfect) measure of risk.

National Health Service

The second dataset I found is from a meta-analysis looking at the cost effectiveness of public health interventions within the English National Health Service (NHS) [15]. This dataset is similar to the DCP2 data above because the only measure of uncertainty is the spread of the estimates. Overall, there seems to be a similar but weaker pattern here:

	guidance_topic	comparator	median	num_estimates	spread
53	Smoking cessation—general population: client c...	Background quit rate; no intervention or usua...	20.000000	8.0	2.288330
4	Exercise prescriptions	Advice	12.987013	4.0	7.194245
56	Smoking cessation—general population: recruitm...	Background quit rate; no intervention or advice	3.846154	15.0	0.074499
62	Smoking cessation —general population: dentist...	Usual care	3.311258	3.0	10.989011
51	Smoking cessation—general population: incentiv...	Intervention no NRT	2.793296	2.0	1.597444
2	BA (5 min plus self-help)	Background quit rate	2.702703	8.0	1.801802
54	Smoking cessation—general population: proactiv...	Usual care or intervention but no telephone c...	2.341920	9.0	0.683527
58	Smoking cessation—general population: identify...	No intervention	1.984127	4.0	0.243902
61	Smoking cessation—general population: pharmaci...	Usual care	1.831502	2.0	4.608295
0	BA only (5 min)	Background quit rate	1.366120	8.0	0.909091
46	PA counselling	No intervention	1.157407	2.0	1.353180
64	Smoking cessation—disadvantaged groups: client...	No intervention	0.639386	3.0	0.166889
1	BA [5 min plus nicotine replacement therapy (N...	Background quit rate	0.473934	8.0	0.315557
70	Smoking cessation—disadvantaged groups: NHS SSS	No intervention	0.372301	2.0	3.311258
71	Smoking cessation—disadvantaged groups: pharma...	No intervention	0.317360	2.0	0.235738
90	Screening and BA during GP consultation	No intervention	0.303030	3.0	0.151515
19	Life-skills training	Normal education	0.286369	3.0	0.180180
74	Statins—disadvantaged groups: invitation for s...	Usual care or no intervention	0.230097	2.0	1.445087
72	Statins—general population: pharmacist based	Usual care or no intervention	0.204415	4.0	0.151837
86	Individual stress management	No intervention	0.200080	3.0	0.086498
87	Curricular	No intervention or standard education	0.138889	4.0	0.093721
30	Urban trail	No intervention	0.095740	4.0	0.044425
13	Brief counselling	Didactic messages	0.082008	2.0	4.385965
11	Accelerated partner therapy—doxycycline	Patient referral	0.071301	2.0	0.106952
15	Information motivation and behaviour skills	Didactic information	0.070706	2.0	0.129634
12	Accelerated partner therapy—azithromycin	Patient referral	0.051480	2.0	0.077220
76	Advice about PA	Usual care	0.027855	2.0	0.051546
16	Enhanced counselling	Didactic messages	0.021927	2.0	0.083243

Global Health Cost Effectiveness Analysis Registry

Finally, I look at a massive dataset from the Global Health Cost Effectiveness Analysis Registry (GHCEA). First, I look at the full distribution of cost effectiveness estimates. It’s pretty clear they’re lognormally distributed:

Next, I look at just the estimates that have confidence intervals, which ends up being about 900 out of the 5000 original estimates. The confidence interval column was too erratic, so I went through by hand and found the upper and lower bound. I also filtered out studies with an overall quality rating below 4.5 (on a 1-7 scale). The end result is 653 estimates that have confidence intervals. Here’s a histogram of the filtered results – note that there’s a hole in the distribution where some of the estimates were filtered out:

Finally, here’s a scatterplot. It’s not the prettiest relationship in the world, but it does seem to have an upward slope:

Evidence from Philanthropy

GiveWell is an organization that does in-depth charity evaluations, often using cost effectiveness estimates in their decision process. They’ve recently changed their approach to explicitly accommodate different philosophical positions, but the older models had their staff estimate different parameters for direct input.

Dan Wahl had the good idea to run a monte carlo simulation by sampling from these staff parameters, which results in a set of estimates you can use to calculate the standard deviation and downside risk for an intervention. I downloaded his code and put the combined outputs into gw_data.csv (see scrape.py), which I include below.

The cost effectiveness rankings here follow a similar pattern to the other datasets, although it’s a little less pronounced:

So if you prefer to use the standard deviation as a measure, there do seem to be returns to risk taking – higher impact estimates tend to be noisier. But if the downside risk metric makes more sense to you, the highest impact estimates underperform the mean less often, so there’s not a return to this measure of risk.

Evidence from Scientific Research

I have two sources of data on the impact of scientific research. The first is from the Future of Humanity Institute’s (FHI) research looking at the long term impact of neglected tropical disease research. The second is data I collected from Google Scholar on the variation in citation counts vs. mean citation counts for individual researchers.

I also found a few related papers in the existing “Science of Science” literature, and summarize those at the end.

FHI Estimates

These numbers differ from the GiveWell numbers above because they are estimates of the value of scientific research, and aren’t derived from randomized control trials of existing treatments. This means we should be much more uncertain about this model and the inputs.

Again, these numbers follow the patterns of earlier estimates with some research topics substantially outperforming others:

So while there is a strong positive relationship between uncertainty and impact, there is a weaker negative relationship between downside risk and impact.

Research Citation Counts

Next, I thought it would be interesting to see if these patterns appear in researcher citation counts. I found a list of ecology researchers along with links to their Google Scholar profiles on GitHub. I treated this list as a population of researchers (I’m not sure if it really is), then randomly selected 100 non-students and downloaded their list of publications and citation counts. I then calculated the mean citation count, standard deviation, and downside risk for each researcher.

The assumption here is that citation count is proportional to real world impact. Another thing to mention is that these scientists have different funding levels, so we don’t know the true funding to citation conversion rate.

Using the standard deviation in a situation like this doesn’t make a lot of sense. By default, a very successful researcher might have a higher standard deviation in their citations as they progress through their career [10]. I think the downside risk metric is more useful here, and it shows that highly cited researchers outperform the mean researcher more often.

Uncertainty in Peer Review

An alternate way to look at this question would be to try to relate peer reviewer uncertainty with eventual citation counts. At least theoretically, it could be rational to fund a study with a lower mean reviewer score if there is sufficient uncertainty [11]. While some research has found a positive relationship between mean reviewer score and eventual citation counts [12], and others have studied the variation in reviewer scores [13], nobody has related the variation in reviewer scores with eventual citation counts. I contacted the NIH and they don’t keep a record of individual reviewer scores for privacy reasons, so this type of study doesn’t seem possible currently.

Surprise as Risk

Another fascinating study takes a different approach by measuring if the subject matter of the paper is risky/surprising [14]. They do this by comparing the chemicals discussed in the paper with an existing network of chemical knowledge. Studies that propose a new type of connection or a jump to new knowledge are judged to be more risky (Figure 1), and are eventually associated with higher citation counts and more scientific awards (Figure 3).

The effect size in this paper isn’t huge – a research strategy that is an order of magnitude less probable receives 2.26 more citations on average. But I think this paper gets closer to measuring the concept of scientific risk than anything else. They also conclude that a scientist trying to maximize citation count would probably focus on repeat projects, so specific policies to encourage higher risk science might be needed.

Conclusion

It’s interesting to see some common patterns emerge across these different domains and datasets.

• First, the impact distributions make it clear that some interventions are much better than others. As a result, it makes sense to spend a lot of time searching for good opportunities.
• Second, interventions with a high downside risk tend to have lower expected impacts. Even though high impact interventions are more uncertain, they dip below the mean less often or to a lesser extent. This is actually a good thing because it means there isn’t a huge downside to pursuing the high impact opportunities, at least in this dataset.
• Third, there do seem to be returns to risk (uncertainty), so a large error bound on a cost effectiveness estimate shouldn’t be disqualifying on it’s own.

Whether or not there are returns to risk, then, depends on your definition of risk. Using the definitions from the introduction, there do seem to be returns to risk (uncertainty). In other words, uncertainty is something you might have to learn to live with if you want to have a big effect on the world.

References

[1] What Are Foundations For? Boston Review. http://bostonreview.net/forum/foundations-philanthropy-democracy

[2] Hits-based Giving. Open Philanthropy Project. https://www.openphilanthropy.org/blog/hits-based-giving

[3] Broad market efficiency. GiveWell. https://blog.givewell.org/2013/05/02/broad-market-efficiency/

[4] The Confusion of Risk vs. Uncertainty. The Guesstimate Blog. https://medium.com/guesstimate-blog/the-confusion-of-risk-vs-uncertainty-1c6cd512aa69

[5] Benefit-Cost Results. Washington State Institute for Public Policy. http://www.wsipp.wa.gov/BenefitCost

[6] Disease Control Priorities in Developing Countries (DCP2). http://www.dcp-3.org/dcp2

[7] GiveWell’s Cost-Effectiveness Analyses. GiveWell. https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness/cost-effectiveness-models

[8] Stochastic Altruism. https://danwahl.github.io/stochastic-altruism

[9] Uncertainty Quantification. Wikipedia. https://en.wikipedia.org/wiki/Uncertainty_quantification#Sources_of_uncertainty

[10] Quantifying the evolution of individual scientific impact. http://science.sciencemag.org/content/354/6312/aaf5239

[11] Improving the Peer review process: Capturing more information and enabling high-risk/high-return research. https://www.sciencedirect.com/science/article/pii/S0048733316301111

[12] Big names or big ideas: Do peer-review panels select the best science proposals? http://science.sciencemag.org/content/348/6233/434

[13] Peer Review Evaluation Process of Marie Curie Actions under EU’s Seventh Framework Programme for Research. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4488366/

[14] Tradition and Innovation in Scientists’ Research Strategies. http://journals.sagepub.com/doi/abs/10.1177/0003122415601618

[15] The cost-effectiveness of public health interventions. Journal of Public Health. https://academic.oup.com/jpubhealth/article/34/1/37/1554654

[16] The Global Health Cost Effectiveness Analysis Registry Tufts University. http://healtheconomics.tuftsmedicalcenter.org/ghcearegistry/