Wednesday, May 09, 2012
Fooled by randomness?
Wikipedia tells me Philip Tetlock is a distinguished chaired professor at Penn, who is best known for being highly skeptical of the value of "expert" forecasts.
Yet today here is a guest post up at Marginal Revolution from him this morning that I find extremely confusing. Tetlock is describing a multi-year forecasting experiment and doing things that I find quite surprising/shocking.
First consider the claim is that the forecasters did well because an ex-post weighted average of the "most insightful and engaged forecasters" crushed a control group.
In fact, they did so well that the Tetlock's group is breaking their planned experimental study after one year and is creating a group of "super-forecasters" based on their first year track record.
People, in a small sample stochastic environment, there will always be individuals that do better than others. This could purely be by chance. If you ex-post weight them higher, you can get a very impressive short term performance.
Generally speaking though, godding them up and following their forecasts into the future is a fools game, because their success was just short term randomness, or in plainer terms, luck.
Or as Mrs. Angus succinctly put it: "mean reversion, y'all".
If you do decided to enter the contest, you may find this information helpful.