Baffle them with bulls**t

Oh my.  Jeff Sachs has a piece in The Lancet, extolling the accomplishments of his Millenium Village Project, that is, shall we say, weak.

Let's concentrate on child mortality, described in the Lancet piece as the "primary study outcome".

Sachs et. al. claim that, "Mortality rates in children younger than 5 years of age decreased by 22% in Millennium Village sites relative to baseline (absolute decrease 25 deaths per 1000 livebirths, p=0·015) and 32% relative to matched comparison sites (30 deaths per 1000 livebirths, p=0·033)."

They present this evidence in a table (page 7 of the article, the last row of the results there) comparing the change from year zero to year 3 in the MVs and in the comparison villages.

So far, so good, right? Well, what if I told you that, in the words of Sachs et. al., "Local comparison village sites were introduced in the third study year to enhance the plausibility that recorded changes were the result of intervention exposure."

In other words, the comparison villages were added ex-post! So how then can they report the change in child mortality in the comparison villages over a time period that started before they started to study those villages?

"Year-0 value is based on recall items in the year 3 survey (eg, women’s reproductive histories)."

YIKES!!

So in year three, they asked women how many children they had under 5 that died in year zero and compared that to the actual numbers they measured in year 3, while the MV change was computed from measurements (not surveys) in both years.

Why does this matter? Well according to the table in the Sachs et. al. article, child mortality is RISING in the comparison villages. This "result" is driving the whole claim quoted above. In the MV case, child mortality falls from 113 to 88.7 over the time period, while in the comparison villages the rate rises from 90 to 96!

So if that comparison is no good what can we use? Well, the total percentage decline in child mortality in the MVs was around 22% which is pretty much right in line with the overall figures the World Bank is reporting for many Sub-Saharan African countries!

I am by far not the first one to point this out. Here's an article from Nature, quoting Michael Clemens than makes many of the same points.. Here's the Roving Bandit pointing out other statistical silliness in the Lancet paper. Here's more from Aid Thoughts about the overall decline of child mortality in Africa.

I really don't understand why a well published PhD economist would bend good practice to this extent, no matter how noble his ultimate goals.