A big trend in economics these days is the Randomized Control Trial (RCT), often referred to by its enthusiasts as the "gold standard" of evidence. An intervention is tested by randomly assigning it across a chosen sample and computing the average treatment effect. An analogy is often made to clinical trials in medicine, but there is (at least) one important difference. Most drug trials are "double blind", where participants do not know if they are in the "control" group or the "treatment" group. By contrast, most RCTs are not double blind. Participants know what group they are in.
And, as a fascinating new paper points out, this knowledge can produce "pseudo-placebo effects" in RCTs. That is to say, the expectation of receiving the treatment can cause people to modify their behaviors in a way that produces a significant "average treatment effect" even if the actual intervention in not particularly effective.
The paper illustrates the point by undertaking two different RCTs on cowpea seeds in Tanzania. One is a traditional study where the control group knows they are getting traditional seeds and the treatment group knows they are getting modern seeds. The second is double blind; neither group knows what seed it is getting.
The traditional RCT shows a significant over 20% increase in yields from the modern seed. But the double blind RCT shows that virtually all of that improvement comes from changed behavior, not from any inherent effectiveness of the modern seed.
Specifically, the average treatment effect in the double blind RCT was zero! And when the harvests in the control groups across the two RCTs were compared, the blind control group showed a significant over 20% increase over the traditional RCT control group which knew they were getting the traditional seeds. This is the "pseudo-placebo" effect and it explains the entire average treatment effect in the traditional RCT.
In other words, the significant effect found in the traditional RCT was not due to better seeds, it was due to actions taken by the farmers who thought they were getting better seeds (they planted them in larger plots with more space between the plants on better quality land). These farmers' expectations were wrong (in post experiment surveys, over 60% of them said they were disappointed in the yields), and the significant effect in the traditional RCT would not survive over time because the farmers, having adjusted their expectations downward would stop taking the actions that produced the "success".
People, can I get a "YIKES"?
Hat tip to Justin Sandefur.