The Social Returns to Risk Taking

If you’re trying to improve the world, should you avoid uncertainty or embrace it? Is it better to spend money on a temporary health intervention or fund research to eventually find a cure? I tried to answer these questions in a previous post by looking at standalone data from a variety of sources. Some of the sources shared similar enough units that they could be combined, so I try to do so below.

All the code and data for this project are available on GitHub here.

Data Wrangling

Because both the GiveWell and Future of Humanity Institute (FHI) data share the same units, I can combine them to get a sense of the scale. Also, if I lazily assume that I can estimate the standard deviation of the Disease Control Priorities (DCP2) and National Institute for Health and Care Excellence (NICE) estimates by dividing the range of the estimates by two, those can be included as well. Note also that the NICE estimates are median results while Givewell and FHI are mean results, so this might need to be changed. There are more in-depth descriptions of the sources for these data in my previous post.

All the other conversions are pretty straightforward, but the Global Health Cost Effectiveness Analysis Registry (GHCEA) data required more wrangling. First, I needed to convert the confidence intervals to standard deviations, which requires making a bunch of assumptions. Here’s what the Cochrane Review has to say about this process:

Confidence intervals for means can also be used to calculate standard deviations . . . Most confidence intervals are 95% confidence intervals. If the sample size is large (say bigger than 100 in each group), the 95% confidence interval is 3.92 standard errors wide (3.92 = 2 × 1.96). The standard deviation for each group is obtained by dividing the length of the confidence interval by 3.92, and then multiplying by the square root of the sample size:
    SD = np.sqrt(N) * (upper_limit - lower_limit)/3.92
For 90% confidence intervals 3.92 should be replaced by 3.29, and for 99% confidence intervals it should be replaced by 5.15.

So if I assume a 95% confidence intervals unless otherwise stated, N=1000 simulations for their monte carlo analyses, and simulated samples that follow a normal distribution, I can back-calculate the standard deviation. This assumes quite a lot (especially the N=1000 simulations part), so I’d be open to advice on how to better handle this. In addition to the calculations above, I filter out any GHCEA studies that were rated below 4.5 by reviewers on a 1-7 quality scale. The end result is 653 interventions with standard deviations and cost effectiveness estimates.

One final thing to consider: the Givewell estimates are from shovel-ready charities accepting donations, while many of the others measure impact against a counterfactual using an Incremental Cost Effectiveness Ratio without a clear avenue for donors to make that change happen. Instead, many of these interventions probably need to be implemented at the hospital, insurer, or government policy level rather than through a charity.

Putting It All Together

Below is a table of the combined 761 estimates and some histograms to get an idea of their distributions (mostly lognormal). Next, I fit some curves, run the portfolio optimization, and visualize the results.

intervention cost_effectiveness stdev source
3 Research - Diarrhoeal diseases 3549.783221 286578.696559 FHI
14 Research - Meningititis 1732.527683 150346.122518 FHI
11 Research - Leishmaniasis 1703.723172 191421.556624 FHI
846 Routine measles-containing vaccine followed by... 1000.000000 5703.964033 GHCEA
413 Aspirin alone (325 mg initial dose & subsequen... 1000.000000 12879.918785 GHCEA
571 Preventive treatment of malaria in pregnancy w... 1000.000000 104.797239 GHCEA
564 Preventive treatment of malaria in pregnancy w... 1000.000000 2618.212999 GHCEA
921 New tuberculosis vaccine (40% efficacy) 1000.000000 7394.027451 GHCEA
17 Research - Leprosy 824.521890 104639.483899 FHI
13 Research - Trypanosomiasis 802.785665 107344.569326 FHI
1 Research - Malaria 801.655755 67817.253156 FHI
16 Research - Multiple salmonella infections 753.616047 46757.225910 FHI
15 Research - Typhoid and paratyphoid fever 709.092672 53697.852674 FHI
12 Research - Chagas disease 534.967344 61260.904879 FHI
514 Syphilis screening before third trimester + tr... 500.000000 2402.946550 GHCEA
887 Brief smoking cessation advice + Bupropion 333.333333 0.461113 GHCEA
912 Syphilis screening before third trimester + tr... 333.333333 1597.109929 GHCEA
338 Intermittent preventive treatment in infants (... 333.333333 2001.390889 GHCEA
523 Syphilis screening before third trimester + tr... 333.333333 1373.112314 GHCEA
786 Intermittent preventive treatment in infants (... 333.333333 2584.320274 GHCEA
0 Research - HIV 303.678832 23736.173393 FHI
6 Research - Trichuriasis 251.336051 11972.794952 FHI
918 New tuberculosis vaccine (40% efficacy) 250.000000 1038.432984 GHCEA
572 Syphilis screening before third trimester + tr... 250.000000 998.193706 GHCEA
771 Syphilis screening before third trimester + tr... 250.000000 1201.473275 GHCEA
294 Syphilis screening before third trimester + tr... 250.000000 1373.112314 GHCEA
789 Malaria intermittent preventive treatment in i... 250.000000 753.353740 GHCEA
327 Syphilis screening before third trimester + tr... 250.000000 1201.473275 GHCEA
634 Magnesium sulfate 250.000000 5703.964033 GHCEA
332 Intermittent preventive treatment in infants (... 250.000000 1786.476431 GHCEA
... ... ... ... ...
19 Risperidone (5 mg/day) 0.025000 96.211442 GHCEA
230 Universal hearing screening using otoacoustic ... 0.024390 0.185710 GHCEA
169 Cochlear implant 0.023810 0.222216 GHCEA
41 HMG-CoA reductase inhibitors (statins) via mai... 0.019608 0.362980 GHCEA
47 Treatment of Kaposi's sarcoma 0.019066 0.014301 DCP2
29 Active After-school Communities Program (AASC)... 0.016949 0.063884 GHCEA
232 Pulse oximetry + clinical assessment 0.016949 0.457310 GHCEA
126 General practitioner referral to exercise phys... 0.015385 0.070669 GHCEA
60 Pre-diabetes screen plus Metformin, diet and e... 0.014286 0.068841 GHCEA
17 Therapy with Bupropion for Tobacco Cessation 0.014286 2.586740 GHCEA
80 Olanzapine (15 mg/day) 0.013158 0.068253 GHCEA
233 Colonoscopy screening every 10 years 0.012987 0.964205 GHCEA
73 Pre-diabetes screen plus Orlistat to prevent o... 0.011494 0.221821 GHCEA
30 Residential treatment + naltrexone 0.010000 0.092855 GHCEA
246 Cochlear implant 0.010000 0.364138 GHCEA
274 Longer individual intervention by telephone fo... 0.010000 0.313466 GHCEA
111 Optimal treatment with antipsychotic medicatio... 0.010000 0.054807 GHCEA
107 Cognitive dissonance – school based bulimia an... 0.009091 0.084410 GHCEA
87 Sibutramine treatment 0.008333 0.088728 GHCEA
681 Cochlear implant 0.008333 0.182431 GHCEA
352 Cochlear implant 0.006250 0.249892 GHCEA
217 Human papilloma virus (HPV) vaccine; 70% coverage 0.006250 0.063415 GHCEA
59 Residential treatment - alcohol 0.005882 0.058717 GHCEA
43 Looma Healthy Lifestyle- community based inter... 0.005556 0.038026 GHCEA
289 Cochlear implant 0.005556 0.333370 GHCEA
63 Current treatment: all patients receiving anti... 0.005263 0.045976 GHCEA
369 Creatinine test conducted at baseline (asympto... 0.005263 0.012386 GHCEA
44 Orlistat treatment 0.004762 0.046974 GHCEA
875 Cochlear implants 0.004167 0.269500 GHCEA
50 Walking School Bus (WSB) program to increase t... 0.001852 0.002584 GHCEA

761 rows × 4 columns

A histogram of all the results together:

And here are histograms of the individual sources:

Fitting Some Curves

So how do I determine if there is a return to risk taking? One approach would be to run a linear regression through the data and see if it has a positive slope. This is what I do first below, but there’s a problem with this approach. To see why, imagine calculating the cost effectiveness of every possible action, including bogus things like lighting $1000 on fire. You’d end up with a lot of useless interventions that would mess up the slope of the linear regression.

So my second approach is to just see if the frontier that encloses the top end of the estimates has a positive slope. In Modern Portfolio Theory, this frontier is called the efficient frontier, which I’ve written about before. I didn’t have enough data to test out this theory in the past, but the combination of all these sources makes it possible to do so now.

An example of an efficient frontier.

Below, I fit a linear regression and a power law to the results. The power law has a slightly higher r-squared value, but this isn’t really a valid measure of goodness of fit for nonlinear curves. The first plot uses standard axes to get a sense of the scale:

Here’s the same plot with log-log axes to get a better view the data:

The Efficient Frontier

Finally, I use a modified version of an algorithm described on Quantopian to generate an efficient frontier. Each point along the curve represents a portfolio of interventions with the highest expected impact for the level of risk. The covariance matrix I used as input is all zeros except for the variances, although this could be changed if you have some reason to think intervention outcomes are correlated in some way.

Note that the plot below is interactive with tooltips and scroll-to-zoom enabled.

Conclusion

  • It seems like there are returns to risk taking for both the individual and combined estimates. This is useful to know because it means a a large error bound on a cost effectiveness estimate shouldn’t be disqualifying on it’s own.
  • Plots like these could be useful for identifying promising interventions, especially when many independent estimates point in the same direction. This seems to be the case for many malaria, HIV, and smoking cessation interventions in the plot above.
  • This framework could also provide a useful sanity check for future estimates. For example, if an estimate is far above the existing frontier, it might be worth reviewing it for an incorrect calculation or poor assumption. But it’s important to be careful when doing this because these estimates only cover a small fraction of the possible actions one could take in the world.
  • The intervention with the highest expected impact (and highest uncertainty) is research into diarrheal disease. This suggests that research can be very beneficial even if it’s more uncertain. This relationship might be even more clear if we were to add estimates from more esoteric forms of basic research, although some forms of research might not be amenable to this type of analysis.

References

[1] Are there returns to risk taking in science, philanthropy, or public policy? https://pstblog.com/2017/12/02/risk-return

[2] Efficient Frontier. Wikipedia. https://en.wikipedia.org/wiki/Efficient_frontier

[3] psthomas: efficient-frontier. GitHub. https://github.com/psthomas/efficient-frontier

[4] Global Health Cost Effectiveness Analysis Registry. Tufts University. http://healtheconomics.tuftsmedicalcenter.org/ghcearegistry/

[5] The Cochrane Review Handbook. http://handbook-5-1.cochrane.org/chapter_7/7_7_3_2_obtaining_standard_deviations_from_standard_errors_and.htm

[6] Relationship between the range and the standard deviation. Stack Exchange. https://stats.stackexchange.com/questions/69575/relationship-between-the-range-and-the-standard-deviation

[7] The Efficient Frontier: Markowitz portfolio optimization in Python. Quantopian.
https://blog.quantopian.com/markowitz-portfolio-optimization-2/

[8] Why Is There No R-Squared for Nonlinear Regression? http://blog.minitab.com/blog/adventures-in-statistics-2/why-is-there-no-r-squared-for-nonlinear-regression