Synthetic Control: Ur (probably) doin' it rong!

People, I've been refereeing a lot of synthetic control papers lately, and I have to say that I don't like what I'm seeing.

I'm seeing 3 big mistakes that people are sometimes trying to sell as features.

Let's discuss.

1. Large, indiscriminate donor pools are not advisable. Look at the Godfather Abadie's papers. The donor pools are 20-40 units. People seem to have the crazy belief that more is better.  It's not.

Let's let Abadie et al. AJPS 2015 explain it:

“Constructing a donor pool of comparison units requires some care. First, units affected by the event or intervention of interest or by events of a similar nature should be excluded from the donor pool. In addition, units that may have suffered large idiosyncratic shocks to the outcome of interest during the study period should also be excluded if such shocks would have not affected the treated unit in the absence of the treatment. Finally, to avoid interpolation biases, it is important to restrict the donor pool to units with characteristics similar to the treated unit. Another reason to restrict the size of the donor pool and consider only units similar to the treated unit is to avoid overfitting. Overfitting arises when the characteristics of the unit affected by the intervention or event of interest are artificially matched by combining idiosyncratic variations in a large sample of unaffected units. "

Got that? pick your donor pools with smarts and with care.


2. Throwing away available pre-intervention outcome data is not advisable.  SC is subject to the same critique as matching, that unobserved factors are not being accounted for. For this reason, the Godfather stresses that the pre-intervention period should be long. From the same AJPS paper:

Critics of Mill’s Method of Differences rightfully point out that the applicability of the method may be limited by the presence of unmeasured factors affecting the out- come variable as well as by heterogeneity in the effects of observed and unobserved factors. However, using a linear factor model, Abadie, Diamond, and Hainmueller (2010) argue that if the number of preintervention periods in the data is large, matching on preintervention outcomes (i.e., on the preintervention counterparts of Y0 and Y1) helps control for unobserved factors and for the heterogene- ity of the effect of the observed and unobserved factors on the outcome of interest. The intuition of this result is straightforward: Only units that are alike in both observed and unobserved determinants of the outcome variable as well as in the effect of those determinants on the outcome variable should produce similar trajectories of the outcome variable over extended periods of time. Once it has been established that the unit representing the case of interest and the synthetic control unit have similar behavior over extended periods of time prior to the intervention, a discrepancy in the outcome variable following the intervention is interpreted as produced by the intervention itself.“ 

I've seen papers discarding pre-intervention data to make the sample "more reasonable". It actually makes the experiment less credible. Now if your are studying a post-Soviet country, sure, you are not going to have a long pre-intervention period. But you should realize that your results are not going to be super robust.

3. Using all the possible lagged outcome variables as predictors is not a good idea! I know, I know, people have done it in good journals and argued in favor of it.

But, "using all outcome lags as separate predictors renders all other covariates irrelevant. This finding holds irrespective of how important these covariates are in order to accurately predict post-treatment values of the outcome, threatening the estimator’s unbiasedness."

To quote Lizzy Warren, "holy guacamole"!!!

Here is a link to the relevant paper.

Here's a bit longer and even scarier quote from it,

"Consequently, in the SCM application we mainly focus on throughout this paper—Billmeier and Nannicini (2013), who analyze the impact of economic liberalization on GDP—the covariates taken from the literature do not affect the synthetic control. The authors obtain the very same counterfactual that would have followed if they had used economically meaningless covariates—or even none at all.3 We further discuss that solely optimizing the pre-treatment fit of the dependent variable and ignoring the covariates can be harmful: the more the covariates are influential for future values of the outcome, the larger a potential bias of the estimated treatment effect will become, possibly leading to wrong conclusions."

So don't use an indiscriminate donor pool. Don't use all the possible lagged outcomes as predictors. Don't throw away pre-intervention data. Unless you want me to go all "reviewer #2" on your asses!