"I shall argue that in ideal circumstances, randomized evaluations of projects are useful for obtaining a convincing estimate of the average effect of a program or project. The price for this success is a focus that is too narrow to tell us “what works” in development, to design policy, or to advance scientific knowledge about development processes. Project evaluation using randomized controlled trials is unlikely to discover the elusive keys to development, nor to be the basis for a cumulative research program that might progressively lead to a better understanding of development."
"This argument applies a fortiori to instrumental variables strategies that are aimed at
generating quasi-experiments; the value of econometric methods cannot be assessed by
how closely they approximate randomized controlled trials. Following Cartwright (2007a, b), I argue that evidence from randomized controlled trials has no special priority. Randomization is not a gold standard because “there is no gold standard,” Cartwright (2007a.) Randomized controlled trials cannot automatically trump other evidence, they do not occupy any special place in some hierarchy of evidence, nor does it make sense to refer to them as “hard” while other methods are “soft”. These rhetorical devices are just that; a metaphor is not an argument."
You can read the whole thing here. Hat tip to The B(l)at(t)man
3 comments:
I tend to agree with the quotes. Randomized trials and instrumental variables are means, not ends. They are good tools, but like any tool they have some problems that call for them and others that don't. I'm not sure why this bothers some researchers.
Best line:
"(H)eterogeneity is not a technical problem calling for an econometric solution, but is a reflection of the fact that we have not started on our proper business, which is trying to understand what is going on."
I cannot explain to you how much--given that I am preparing for a lit review--this rocks my worldview.
Post a Comment