Wednesday, May 09, 2012

Fooled by randomness?

Wikipedia tells me  Philip Tetlock is a distinguished chaired professor at Penn, who is best known for being highly skeptical of the value of "expert" forecasts.

Yet today here is a guest post up at Marginal Revolution from him this morning that I find extremely confusing. Tetlock is describing a multi-year forecasting experiment and doing things that I find quite surprising/shocking.

First consider the claim is that the forecasters did well because an ex-post weighted average of the "most insightful and engaged forecasters" crushed a control group.


In fact, they did so well that the Tetlock's group is breaking their planned experimental study after one year and is creating a group of "super-forecasters" based on their first year track record.

Double Yikes!

People, in a small sample stochastic environment, there will always be individuals that do better than others. This could purely be by chance. If you ex-post weight them higher, you can get a very impressive short term performance.

Generally speaking though, godding them up and following their forecasts into the future is a fools game, because their success was just short term randomness, or in plainer terms, luck.

Or as Mrs. Angus succinctly put it: "mean reversion, y'all".

 If you do decided to enter the contest, you may find this information helpful.


Gerardo said...

He can apply to be Steven Landsburg's "Royal Head Flipper"

SheetWise said...

"... the forecasters did well because an ex-post weighted average of the 'most insightful and engaged forecasters' crushed a control group."

Sounds like survival of the fittest.

Anonymous said...

I sincerely hope that Mrs. Angus does not apply the same comment to your offspring when they misbehave.

visachris327 said...

I come to your post again. It's so great that I like it very much. I also like ray ban prescription sunglasses. I want to introduce oakley sales to you. You can go to know more about cheap monster beats.

Unknown said...

What are you aware about Christian where to get christian louboutin shoes Louboutin shoes? Christian Louboutin would be where to get christian louboutin shoes cheap a France brand.Christian Louboutin shoes are christian louboutin boots easy recognized on earth of trainers christian louboutin pigalle designers given that they have a cheap michael kors handbags variety of colour patterns as well Hermes Plume Bag as a rage among women. Trademark Michael Kors Hamilton of Christian Louboutin lady shoes is red colored leather soles.

visachris322 said...

ugg boots said...

Unser Online-Shop bietet günstige Monster Beats By Dre, Beats By Dre Outlet,Beats Dr Dre Studio Und Dr. Dre Beats Solo-Sale. Hohe Qualität und lLowest Preis,Genießen Sie sparen bis zu 70%.