Tuesday, July 18, 2017

Synthetic Control: Ur (probably) doin' it rong!

People, I've been refereeing a lot of synthetic control papers lately, and I have to say that I don't like what I'm seeing.

I'm seeing 3 big mistakes that people are sometimes trying to sell as features.

Let's discuss.

1. Large, indiscriminate donor pools are not advisable. Look at the Godfather Abadie's papers. The donor pools are 20-40 units. People seem to have the crazy belief that more is better.  It's not.

Let's let Abadie et al. AJPS 2015 explain it:

“Constructing a donor pool of comparison units requires some care. First, units affected by the event or intervention of interest or by events of a similar nature should be excluded from the donor pool. In addition, units that may have suffered large idiosyncratic shocks to the outcome of interest during the study period should also be excluded if such shocks would have not affected the treated unit in the absence of the treatment. Finally, to avoid interpolation biases, it is important to restrict the donor pool to units with characteristics similar to the treated unit. Another reason to restrict the size of the donor pool and consider only units similar to the treated unit is to avoid overfitting. Overfitting arises when the characteristics of the unit affected by the intervention or event of interest are artificially matched by combining idiosyncratic variations in a large sample of unaffected units. "

Got that? pick your donor pools with smarts and with care.

2. Throwing away available pre-intervention outcome data is not advisable.  SC is subject to the same critique as matching, that unobserved factors are not being accounted for. For this reason, the Godfather stresses that the pre-intervention period should be long. From the same AJPS paper:

Critics of Mill’s Method of Differences rightfully point out that the applicability of the method may be limited by the presence of unmeasured factors affecting the out- come variable as well as by heterogeneity in the effects of observed and unobserved factors. However, using a linear factor model, Abadie, Diamond, and Hainmueller (2010) argue that if the number of preintervention periods in the data is large, matching on preintervention outcomes (i.e., on the preintervention counterparts of Y0 and Y1) helps control for unobserved factors and for the heterogene- ity of the effect of the observed and unobserved factors on the outcome of interest. The intuition of this result is straightforward: Only units that are alike in both observed and unobserved determinants of the outcome variable as well as in the effect of those determinants on the outcome variable should produce similar trajectories of the outcome variable over extended periods of time. Once it has been established that the unit representing the case of interest and the synthetic control unit have similar behavior over extended periods of time prior to the intervention, a discrepancy in the outcome variable following the intervention is interpreted as produced by the intervention itself.“ 

I've seen papers discarding pre-intervention data to make the sample "more reasonable". It actually makes the experiment less credible. Now if your are studying a post-Soviet country, sure, you are not going to have a long pre-intervention period. But you should realize that your results are not going to be super robust.

3. Using all the possible lagged outcome variables as predictors is not a good idea! I know, I know, people have done it in good journals and argued in favor of it.

But, "using all outcome lags as separate predictors renders all other covariates irrelevant. This finding holds irrespective of how important these covariates are in order to accurately predict post-treatment values of the outcome, threatening the estimator’s unbiasedness."

To quote Lizzy Warren, "holy guacamole"!!!

Here is a link to the relevant paper.

Here's a bit longer and even scarier quote from it,

"Consequently, in the SCM application we mainly focus on throughout this paper—Billmeier and Nannicini (2013), who analyze the impact of economic liberalization on GDP—the covariates taken from the literature do not affect the synthetic control. The authors obtain the very same counterfactual that would have followed if they had used economically meaningless covariates—or even none at all.3 We further discuss that solely optimizing the pre-treatment fit of the dependent variable and ignoring the covariates can be harmful: the more the covariates are influential for future values of the outcome, the larger a potential bias of the estimated treatment effect will become, possibly leading to wrong conclusions."

So don't use an indiscriminate donor pool. Don't use all the possible lagged outcomes as predictors. Don't throw away pre-intervention data. Unless you want me to go all "reviewer #2" on your asses!

Monday, July 10, 2017

Listen up people, the Body of Christ AIN'T GLUTEN-FREE!

.....But it can contain GMOs.

This is an actual ruling from an actual religion in the actual year or our Lord 2017.

Since neither Tony Gill or Phil Magness are Pope, you may wonder why this bizarre edict has been issued.

Well, "The new rules are needed because the bread is now sold in supermarkets and on the internet, the cardinal said."

LOL, thanks for clearing that up.

So there you have it people. If you want to pretend that a piece of bread turns into a 2000 year old body, you better be damn well sure that bread has gluten. You know, just like Jesus!

Monday, July 03, 2017

NOT an actual email from a book author...

To be quite clear....

I did not receive the following email from any Duke colleague, or any professor of history. But I did receive the email, and I thought I would repost it.  It pretends (falsely) to be a summary of my review of Democracy in Chains. It isn't. But it is fair to say this is what the author might have said.


Thanks so much for the very positive review and affirmation of my book Democracy in Chains by my colleague Duke Professor of Political Science Dr. Michael Munger. Please find below salient extracts from Dr. Munger’s review.

Democracy in Chains: The Deep History of the Radical Right’s Stealth Plan for America, by my Duke University colleague, Nancy MacLean, a professor in our distinguished Department of History… is… a remarkable book. MacLean has argued persuasively throughout her career for the historical method…in this book… MacLean recounts an exchange, a conversation really, between two conservatives…intent on reverse-engineering a …political order in America…using shadowy methods and discredited theories. Democracy in Chains is a work of …historical...research underpinning …facts … from a much larger record…drawing reliable conclusions about history. 

Democracy in Chains is a great story… of … James M. Buchanan, the winner of the 1986 Nobel Prize in Economic Sciences. MacLean is able to decode the true meaning of his …writings which…sought … to bring down.. America and replace it with a plutocracy. …MacLean’s excellence as a writer,…careful sifting of evidence and respectful encounters with opposing points of view…reveal … that …Buchanan … wanted to establish a …society …for racial segregation… 

MacLean’s book…is admirably academic and careful. … MacLean …found …the attempt by segregationist forces to support vouchers. MacLean says, “The economists made their case in the race-neutral, value-free language of their discipline, offering what they depicted as a strictly economic argument—on ‘matters of fact, not values.’” MacLean … support the claim that Buchanan advocated vouchers for the purpose of achieving segregation. … Buchanan’s support for vouchers and for school choice arose from a deeply held concern for …a …repressive apartheid society where African-Americans were …murderous and … must be forcibly suppressed… MacLean has discovered a number of important documents from the history of Public Choice, and other aspects of the history of the 1960s and 1970s in academic economic circles. There is a terrific example on pp. 115–117, where the “glee” of Buchanan and others about their conspiracy, gathered around a roaring fire in the remote mountains of Virginia, is documented.

 … MacLean has…written that history, using … public documents that … destroy...the conspiracy; …that … would sweep the nation, and the world… When summarized in this way, MacLean’s thesis really does read like a … narrative thread connecting the documents and discussions that …strategize about how to win back the White House and rejuvenate the conservative movement… The contribution of Democracy in Chains, then, is to do two things…Identify James Buchanan as the focal point of the revolution, and identify the content of Public Choice research and teaching as anti-Constitutional and anti-democratic… As I hope has been clear, as a book Democracy in Chains is well-written, and the research it contains is both interesting and …illuminating…as an actual history…of the work of James Buchanan …to end democracy in America. 

My thanks to the actual author of this email, Steve Spearman. And he is right: every word of the above actually appears in my review, and in precisely this order!