Fooled by randomness?

Wikipedia tells me  Philip Tetlock is a distinguished chaired professor at Penn, who is best known for being highly skeptical of the value of "expert" forecasts.

Yet today here is a guest post up at Marginal Revolution from him this morning that I find extremely confusing. Tetlock is describing a multi-year forecasting experiment and doing things that I find quite surprising/shocking.

First consider the claim is that the forecasters did well because an ex-post weighted average of the "most insightful and engaged forecasters" crushed a control group.

Yikes!

In fact, they did so well that the Tetlock's group is breaking their planned experimental study after one year and is creating a group of "super-forecasters" based on their first year track record.

Double Yikes!

People, in a small sample stochastic environment, there will always be individuals that do better than others. This could purely be by chance. If you ex-post weight them higher, you can get a very impressive short term performance.

Generally speaking though, godding them up and following their forecasts into the future is a fools game, because their success was just short term randomness, or in plainer terms, luck.

Or as Mrs. Angus succinctly put it: "mean reversion, y'all".

 If you do decided to enter the contest, you may find this information helpful.