If you’re trying to improve the world, should you avoid uncertainty or embrace it? Is it better to spend money on a temporary health intervention or fund research to eventually find a cure? I tried to answer these questions in a previous post by looking at standalone data from a variety of sources. Some of the sources shared similar enough units that they could be combined, so I try to do so below.
All the code and data for this project are available on GitHub here.
Because both the GiveWell and Future of Humanity Institute (FHI) data share the same units, I can combine them to get a sense of the scale. Also, if I lazily assume that I can estimate the standard deviation of the Disease Control Priorities (DCP2) and National Institute for Health and Care Excellence (NICE) estimates by dividing the range of the estimates by two, those can be included as well. Note also that the NICE estimates are median results while Givewell and FHI are mean results, so this might need to be changed. There are more in-depth descriptions of the sources for these data in my previous post.
All the other conversions are pretty straightforward, but the Global Health Cost Effectiveness Analysis Registry (GHCEA) data required more wrangling. First, I needed to convert the confidence intervals to standard deviations, which requires making a bunch of assumptions. Here’s what the Cochrane Review has to say about this process:
Confidence intervals for means can also be used to calculate standard deviations . . . Most confidence intervals are 95% confidence intervals. If the sample size is large (say bigger than 100 in each group), the 95% confidence interval is 3.92 standard errors wide (3.92 = 2 × 1.96). The standard deviation for each group is obtained by dividing the length of the confidence interval by 3.92, and then multiplying by the square root of the sample size:
SD = np.sqrt(N) * (upper_limit - lower_limit)/3.92
For 90% confidence intervals 3.92 should be replaced by 3.29, and for 99% confidence intervals it should be replaced by 5.15.
So if I assume a 95% confidence intervals unless otherwise stated, N=1000 samples from their simulations, and simulated samples that follow a normal distribution, I can back-calculate the standard deviation. This assumes quite a lot (especially the N=1000 samples part), so I’d be open to advice on how to better handle this. In addition to the calculations above, I filter out any GHCEA studies that were rated below 4.5 by reviewers on a 1-7 quality scale. The end result is 653 interventions with standard deviations and cost effectiveness estimates.
One final thing to consider: the Givewell numbers are estimates of the marginal impact of donating to existing charities, while many of the other sources measure impact against a counterfactual using an Incremental Cost Effectiveness Ratio without a clear avenue for donors to make that change happen. Instead, many of these interventions probably need to be implemented at the hospital, insurer, or government policy level rather than through a charity (although a charity could lobby for these changes). So the GiveWell numbers might be more rigorous because they’re estimates of the current scaled-up impact of an intervention, while some of the other estimates might be too optimistic because they come from experiments implemented under ideal circumstances at a certain point in time.
Putting It All Together
Below is a table of the combined 761 estimates and some histograms to get an idea of their distributions (mostly lognormal). Next, I fit some curves, run the portfolio optimization, and visualize the results.
|3||Research - Diarrhoeal diseases||3549.783221||286578.696559||FHI|
|14||Research - Meningititis||1732.527683||150346.122518||FHI|
|11||Research - Leishmaniasis||1703.723172||191421.556624||FHI|
|846||Routine measles-containing vaccine followed by...||1000.000000||5703.964033||GHCEA|
|413||Aspirin alone (325 mg initial dose & subsequen...||1000.000000||12879.918785||GHCEA|
|571||Preventive treatment of malaria in pregnancy w...||1000.000000||104.797239||GHCEA|
|564||Preventive treatment of malaria in pregnancy w...||1000.000000||2618.212999||GHCEA|
|921||New tuberculosis vaccine (40% efficacy)||1000.000000||7394.027451||GHCEA|
|17||Research - Leprosy||824.521890||104639.483899||FHI|
|13||Research - Trypanosomiasis||802.785665||107344.569326||FHI|
|1||Research - Malaria||801.655755||67817.253156||FHI|
|16||Research - Multiple salmonella infections||753.616047||46757.225910||FHI|
|15||Research - Typhoid and paratyphoid fever||709.092672||53697.852674||FHI|
|12||Research - Chagas disease||534.967344||61260.904879||FHI|
|514||Syphilis screening before third trimester + tr...||500.000000||2402.946550||GHCEA|
|887||Brief smoking cessation advice + Bupropion||333.333333||0.461113||GHCEA|
|912||Syphilis screening before third trimester + tr...||333.333333||1597.109929||GHCEA|
|338||Intermittent preventive treatment in infants (...||333.333333||2001.390889||GHCEA|
|523||Syphilis screening before third trimester + tr...||333.333333||1373.112314||GHCEA|
|786||Intermittent preventive treatment in infants (...||333.333333||2584.320274||GHCEA|
|0||Research - HIV||303.678832||23736.173393||FHI|
|6||Research - Trichuriasis||251.336051||11972.794952||FHI|
|918||New tuberculosis vaccine (40% efficacy)||250.000000||1038.432984||GHCEA|
|572||Syphilis screening before third trimester + tr...||250.000000||998.193706||GHCEA|
|771||Syphilis screening before third trimester + tr...||250.000000||1201.473275||GHCEA|
|294||Syphilis screening before third trimester + tr...||250.000000||1373.112314||GHCEA|
|789||Malaria intermittent preventive treatment in i...||250.000000||753.353740||GHCEA|
|327||Syphilis screening before third trimester + tr...||250.000000||1201.473275||GHCEA|
|332||Intermittent preventive treatment in infants (...||250.000000||1786.476431||GHCEA|
|19||Risperidone (5 mg/day)||0.025000||96.211442||GHCEA|
|230||Universal hearing screening using otoacoustic ...||0.024390||0.185710||GHCEA|
|41||HMG-CoA reductase inhibitors (statins) via mai...||0.019608||0.362980||GHCEA|
|47||Treatment of Kaposi's sarcoma||0.019066||0.014301||DCP2|
|29||Active After-school Communities Program (AASC)...||0.016949||0.063884||GHCEA|
|232||Pulse oximetry + clinical assessment||0.016949||0.457310||GHCEA|
|126||General practitioner referral to exercise phys...||0.015385||0.070669||GHCEA|
|60||Pre-diabetes screen plus Metformin, diet and e...||0.014286||0.068841||GHCEA|
|17||Therapy with Bupropion for Tobacco Cessation||0.014286||2.586740||GHCEA|
|80||Olanzapine (15 mg/day)||0.013158||0.068253||GHCEA|
|233||Colonoscopy screening every 10 years||0.012987||0.964205||GHCEA|
|73||Pre-diabetes screen plus Orlistat to prevent o...||0.011494||0.221821||GHCEA|
|30||Residential treatment + naltrexone||0.010000||0.092855||GHCEA|
|274||Longer individual intervention by telephone fo...||0.010000||0.313466||GHCEA|
|111||Optimal treatment with antipsychotic medicatio...||0.010000||0.054807||GHCEA|
|107||Cognitive dissonance – school based bulimia an...||0.009091||0.084410||GHCEA|
|217||Human papilloma virus (HPV) vaccine; 70% coverage||0.006250||0.063415||GHCEA|
|59||Residential treatment - alcohol||0.005882||0.058717||GHCEA|
|43||Looma Healthy Lifestyle- community based inter...||0.005556||0.038026||GHCEA|
|63||Current treatment: all patients receiving anti...||0.005263||0.045976||GHCEA|
|369||Creatinine test conducted at baseline (asympto...||0.005263||0.012386||GHCEA|
|50||Walking School Bus (WSB) program to increase t...||0.001852||0.002584||GHCEA|
761 rows × 4 columns
A histogram of all the results together:
And here are histograms of the individual sources:
Fitting Some Curves
So how do I determine if there is a return to risk taking? One approach would be to run a linear regression through the data and see if it has a positive slope. This is what I do first below, but there’s a problem with this approach. To see why, imagine calculating the cost effectiveness of every possible action, including bogus things like lighting $1000 on fire. You’d end up with a lot of useless interventions that would mess up the slope of the linear regression.
So my second approach is to just see if the frontier that encloses the top end of the estimates has a positive slope. In Modern Portfolio Theory, this frontier is called the efficient frontier, which I’ve written about before. I didn’t have enough data to test out this theory in the past, but the combination of all these sources makes it possible to do so now.
Below, I fit a linear regression and a power law to the results. The power law doesn’t have an r-squared value because this isn’t really a valid measure of goodness of fit for nonlinear curves. The first plot uses standard axes to get a sense of the scale:
Here’s the same plot with log-log axes to get a better view the data:
The Efficient Frontier
Finally, I use a modified version of an algorithm described on Quantopian to generate an efficient frontier. Each point along the curve represents a portfolio of interventions with the highest expected impact for the level of risk. The covariance matrix I used as input is all zeros except for the variances, although this could be changed if you have some reason to think intervention outcomes are correlated in some way.
Note that the plot below is interactive with tooltips and scroll-to-zoom enabled.
- It seems like there are returns to risk taking for both the individual and combined estimates. This is useful to know because it means a a large error bound on a cost effectiveness estimate shouldn’t be disqualifying on it’s own.
- Plots like these could be useful for identifying promising interventions, especially when many independent estimates point in the same direction. This seems to be the case for many malaria, HIV, and smoking cessation interventions in the plot above.
- This framework could also provide a useful sanity check for future estimates. For example, if an estimate is far above the existing frontier, it might be worth reviewing it for an incorrect calculation or poor assumption. But it’s important to be careful when doing this because these estimates only cover a small fraction of the possible actions one could take in the world.
- The intervention with the highest expected impact (and highest uncertainty) is research into diarrheal disease. This suggests that research can be very beneficial even if it’s more uncertain. This relationship might be even more clear if we were to add estimates from more esoteric forms of basic research, although some forms of research might not be amenable to this type of analysis.
 Are there returns to risk taking in science, philanthropy, or public policy? https://pstblog.com/2017/12/02/risk-return
 Efficient Frontier. Wikipedia. https://en.wikipedia.org/wiki/Efficient_frontier
 psthomas: efficient-frontier. GitHub. https://github.com/psthomas/efficient-frontier
 Global Health Cost Effectiveness Analysis Registry. Tufts University. http://healtheconomics.tuftsmedicalcenter.org/ghcearegistry/
 The Cochrane Review Handbook. http://handbook-5-1.cochrane.org/chapter_7/7_7_3_2_obtaining_standard_deviations_from_standard_errors_and.htm
 Relationship between the range and the standard deviation. Stack Exchange. https://stats.stackexchange.com/questions/69575/relationship-between-the-range-and-the-standard-deviation
 The Efficient Frontier: Markowitz portfolio optimization in Python. Quantopian.
 Why Is There No R-Squared for Nonlinear Regression? http://blog.minitab.com/blog/adventures-in-statistics-2/why-is-there-no-r-squared-for-nonlinear-regression