This is a follow up exploring the data that I collected in an earlier post. All of the data is from the IGM Experts Forum, which surveys a group of leading economists on a variety of policy questions. A CSV of all the data is available here, as are separate datasets of the questions and responses. An IPython notebook with all of the code from this analysis is available here.
I’m especially interested in how confidence changes with the scale of a claim, so I use a few different techniques to look at that relationship. First, I look at confidence by vote type and find that economists seem to be more confident when they
strongly agree or
strongly disagree. Second, I find that confidence actually increases the further a view is from the median, although this is relationship is mainly driven by
25 votes out of a
7024 vote sample.
An earlier paper [1, PDF] by Gordon and Dahl found that male economists and economists that were educated at the University of Chicago and MIT seemed to be more confident. I find less evidence of this in the newer data, although I lack the knowledge of statistics to say whether any of these differences are significant.
The main takeaway from this analysis is the amazing amount of consensus among leading economists. The mean and median distance away from the consensus responses are
0.45 points on a five point scale. Roughly
90 percent of the responses are within
1.5 units of the consensus for all questions. These results are consistend with Gordon and Dahl’s earlier findings [2, URL].
Update: A recent study  took a look at these data and they make a few important points. First, if the intention of the forum is to show consensus in the economics profession, this might introduce a selection bias towards non-controversial questions. Second, they do find evidence of institutional and political bias but suggest it is a result of the hiring process rather than the educational process.
First, let’s look at some descriptive statistics for the responses:
Confidence Grouped by Vote Type
Next, I use the Pandas dataframe grouping function to look at confidence by vote type. Note that the
Strongly Disagree votes are much less common.
Agree is the most common vote, followed by
Uncertain has the lowest mean confidence at
4.3, possibly because it’s weird to say that you’re ‘confidently uncertain’.
Strongly Agree and
Strongly Disagree have higher mean and median confidences. One possible explanation is that economists are unwilling to step into the
Strongly categories unless they feel that they have very good evidence. Another possibility is that this is an issue with the survey – it’s weird to say that you ‘unconfidently strongly agree’.
Vote Distance from the Median
The above results are interesting, but what I’m more interested in is confidence as a claim becomes more controversial. Below, I construct a measure of vote distance from the median vote, then look at confidence grouped by distance from that median. I use the Pandas
apply function below to assign a value ranging from 0 (
Strongly Disagree) to 4 (
Strongly Agree) to both the vote and median_vote columns. I then take the absolute value of the difference between each vote and the median_vote for each question to calculate the distance.
Confidence does increase the further the vote is from the median view, but relationship this is driven by
385 votes two points away, and
25 votes three points away out of a
7024 vote sample. It’s possible that these confident yet controversial votes are from subject matter experts and have more information about a topic than the rest.
Making a Continuous Vote Column
To add some granularity to the above data, I combine the vote number column and the confidence column into one incremental column called
incr_votenum. So a vote of
Agree (vote_num = 3) at a confidence of
5 leads to a
3 + 5/11). The assumption I am making here is that confidence is a continuous measure between two votes, with an
Agree vote of confidence
10 measuring less than a
Strongly Agree vote at confidence
0. I’m not sure if this is a safe assumption to make, but I’m going to run with it.
I then calculate the median incr_votenum for each question, and the distance away from the median for each vote. A few example results are shown in the table below.
|0||Brexit II||Question A||2||4||2.363636||3.363636||1.000000|
|1||Brexit II||Question B||3||5||3.454545||3.272727||0.181818|
|198||Brexit II||Question A||3||5||3.454545||3.363636||0.090909|
|199||Brexit II||Question B||3||5||3.454545||3.272727||0.181818|
|397||Brexit II||Question A||3||3||3.272727||3.363636||0.090909|
Visualizing the Spread of the Votes
The following boxplot shows the distance from the median for all votes, using the new incremental vote measure. It’s pretty amazing that the mean and median distance away from the consensus are only
0.628 respectively. That’s an impressive amount of consensus. The whiskers on the boxplot cover the 90 percent of the data that fall within roughly 1.6 points of the consensus vote on a scale from 0 to 5.
The histogram also shows a suprising amount of consensus, although it also shows a second peak around a difference of 1.0, which is the difference between two bordering answers, e.g. the distance from
Median: 0.454545454545 Mean: 0.628856906192 Stdev: 0.556692286667
Which questions are most controversial?
As a measure of how controversial a question is, I take the standard deviation of the incremental vote number (incr_votenum). I include the final boxplot below, showing the spread of the votes by question.
Which questions are least controversial?
Which economists give more controversial responses?
Which economists give the least controversial responses?
Do any institutions give more controversial responses?
Here’s what Gordon and Dahl had to say about differences between institutions using the 2012 question sample [1, PDF]:
Respondents are dramatically more confident when the academic literature on the topic is large. Not surprisingly, experts on a subject are much more confident about their answers. The middle-aged cohort (the one closest to the current literature) is the most confident, while the oldest (and wisest) cohort is the least confident. Men and those who have worked in Washington do show some tendency to be more confident. Respondents who got their degrees at Chicago are far more confident than the other respondents, with almost as strong an effect for respondents with PhDs from MIT and to a lesser extent from Harvard. Respondents now employed at Yale and to a lesser degree Princeton, MIT, and Stanford seem to be more confident.
It doesn’t seem like any institution sticks out based on this newer data, but maybe with more advanced statistical techniques it might be possible to find something significant.
Are any institutions more confident than others?
Again, although there are differences in the mean distance from the consensus view, all the standard deviations overlap, so I don’t think there is anything significant here. Note that Gordon and Dahl also looked at where economists where educated, rather than just where they were employed, and found differences in confidence based on that metric.
Are male economists more confident?
Gordon and Dahl (2012) noted more confidence among male economists:
The only statistically significant deviation from homogeneous views, therefore, is less caution among men in expressing an opinion, perhaps due to a greater “expert bias.” Personality differences rather than different readings of the existing evidence would then explain these gender effects.
This relationship seems to be less obvious with this expanded dataset. I’m not re-creating their analysis, though, so the difference might still be there if I were to use the controls that they do.
This is just a first look at the data, and overall there seem to be some pretty interesting patterns. There are probably some other interesting ways to augment this data with other information, like which institution an economist studied at, so I might do that in the future.
Feel free to use the data, let me know what you find!
 Views among Economists: Professional Consensus or Point-Counterpoint? Roger Gordon and Gordon B. Dahl. http://econweb.ucsd.edu/~gdahl/papers/views-among-economists.pdf
 Consensus, Polarization, and Alignment in the Economics Profession. Tod S. Van Gunten, John Levi Martin, Misha Teplitskiy. https://www.sociologicalscience.com/articles-v3-45-1028/