Wednesday, May 09, 2012

Fooled by randomness?



Wikipedia tells me  Philip Tetlock is a distinguished chaired professor at Penn, who is best known for being highly skeptical of the value of "expert" forecasts.

Yet today here is a guest post up at Marginal Revolution from him this morning that I find extremely confusing. Tetlock is describing a multi-year forecasting experiment and doing things that I find quite surprising/shocking.

First consider the claim is that the forecasters did well because an ex-post weighted average of the "most insightful and engaged forecasters" crushed a control group.

Yikes!

In fact, they did so well that the Tetlock's group is breaking their planned experimental study after one year and is creating a group of "super-forecasters" based on their first year track record.

Double Yikes!

People, in a small sample stochastic environment, there will always be individuals that do better than others. This could purely be by chance. If you ex-post weight them higher, you can get a very impressive short term performance.

Generally speaking though, godding them up and following their forecasts into the future is a fools game, because their success was just short term randomness, or in plainer terms, luck.

Or as Mrs. Angus succinctly put it: "mean reversion, y'all".

 If you do decided to enter the contest, you may find this information helpful.


5 comments:

Gerardo said...

He can apply to be Steven Landsburg's "Royal Head Flipper"

http://books.google.com/books?id=_aZmzdlrBZMC&lpg=PA309&ots=QAVbX0KqHy&dq=the%20royal%20head%20flipper&pg=PA309#v=onepage&q&f=false

SheetWise said...

"... the forecasters did well because an ex-post weighted average of the 'most insightful and engaged forecasters' crushed a control group."

Sounds like survival of the fittest.

Anonymous said...

I sincerely hope that Mrs. Angus does not apply the same comment to your offspring when they misbehave.

visachris322 said...

http://www.patriotsjerseysprosale.com/aaron-hernandez-jersey/aaron-hernandez-jersey-new-england-patriots-black-shadow-81-jersey.html

http://www.patriotsjerseysprosale.com/andre-carter-jersey-/andre-carter-jersey-new-england-patriots-blue-93-jersey.html

http://www.patriotsjerseysprosale.com/andre-tippett-jersey/andre-tippett-throwback-jersey-new-england-patriots-mitchell-and-red-56-ness-nfl-authenticjersey.html

http://www.patriotsjerseysprosale.com/benjarvus-green-ellis-jersey/reebok-benjarvus-green-ellis-jersey-new-england-patriots-black-shadow-42-nfl-authentic-jersey.html

Likewhywer said...


Unser Online-Shop bietet günstige Monster Beats By Dre, Beats By Dre Outlet,Beats Dr Dre Studio Und Dr. Dre Beats Solo-Sale. Hohe Qualität und lLowest Preis,Genießen Sie sparen bis zu 70%.

http://www.beatsbydrebilligkaufen.net/

http://www.clarisonicdeutschland.com/