Politics and RCTs

Justin Sandefur and longtime KPC friend Mwangi Kimenyi along with Tessa Bold, Germano Mwabu & Alice Ng’ang’a  have written a remarkable paper about the non-uniform results of an educational intervention in Kenya. The paper is well-deserving of discussion, but so is the story of its evolution.

The paper studies an intervention that adds "contract teachers" to schools. Contract teachers are meant to be teachers outside of the main educational bureaucracy who in some way have close ties and more accountability to the local community than the "regular" teachers. In the study, some of the intervention was run by the government, and some was run by an NGO (Worldvision). Test scores in math and reading went up by 0.2 standard deviations compared to the control schools when the intervention was run by the NGO and this increase was statistically significant. However, the intervention had no effect on test scores when it was administered by the government.

This result alone points out the difficulties involved in scaling up education intervention that have been tested by RCTs run by NGOs. Size means government and government might not work.

But people, there is so much more to the story!

The concept of contract teachers initially involved remedial teaching. Banerjee, Cole, Duflo and Linden (QJE 2007) study an NGO-run program in India where the contract teachers tutored remedial students (which raised test scores 0.28 standard deviations). Duflo, Duplass, & Kremer study a contract teacher RCT in Kenya that included the concept of "tracking" where contract teachers were added to a specific class. In some cases the class was randomly split into two groups; in others it was split into low and high scorers on an initial test. This split into more homogeneous classes produced the biggest positive results in the trial.

In an email exchange, Justin told me that while Duflo encouraged him to include a tracking component in his study, she said that it was very unpopular and hard to administer. It is also hard to imagine a government run program that would allow such a component. Think about the USA. What would parents do if they found that classes were being segregated by test scores and their kid was in the "dumb" group?

Because they were explicitly interested in the idea of scaling up a program that could be run by the government, Sandefur et. al. did not include any idea of tracking in their study. In other words, they judged a key element of the success of contract teachers in previous RCTs to be politically unviable ex-ante.

But there's more!

The Sandefur study was part of a pilot program in Kenya. However, things didn't go according to plan:

the Ministry opted to scale-up the contract teacher program before the pilot was completed. Thus the randomized pilot program analyzed here was launched in June 2010, and in October 2010 the Ministry hired 18,000 contract teachers nationwide, nearly equivalent to one per school. These 18,000 teachers were initially hired on two-year, non-renewable contracts, at salary levels of roughly $135 per month, somewhat higher than the highest tier for the pilot phase. In 2011 the Ministry succumbed to political pressure and agreed to allow the contract teachers to unionize and subsequently to hire all 18,000 contract teachers into the civil service at the end of their contracts.

In other words, 18,000 supposed "intervention" teachers became "control" teachers! In plainer terms, they switched from being part of the solution to being part of the problem. Although maybe not, because as Sandefur et. al showed, the government administered contract teachers had no positive impact on outcomes.

In sum, the Sandefur et. al paper shows that while small scale contract teacher RCTs produced modest but positive results, it is not likely those results will survive scaling and government administration.

So what to do? Well Justin & Mwangi along with Tessa Bold and Germano Mwabu have another paper that points to what I believe is the solution at least in the short and medium term. They show that in Kenya, being in a private school raises test scores by one full standard deviation relative to public schools, other relevant factors held constant (this is not an RCT but rather uses "observational" data).

So on the one hand we have these interventions in public schools that raise outcomes by a couple tenths of a standard deviation when implemented on a small scale by NGOs and that may will have no effect when scaled up and implemented by governments.

On the other hand we have an institution (private schools) that raises test scores dramatically more by effectively solving the teacher accountability problems that seem to be behind the outcome problems in public schools in Kenya and other developing countries.

Let me channel Milton Friedman and James Tooley and suggest expanded private schooling with a public voucher program as potentially the greatest pay-off educational intervention available in such situations.