Monday, September 28, 2015

Friends don't let Friends do IV

Just don't do it. And if you must do it, dear God please don't do it with a Arellano-Bond type dynamic panel model (it's the worst, Jerry).

Here are the problems.

First of all, no matter what you may have read or been taught, identification is always and everywhere an ASSUMPTION. You cannot prove your IV is valid.

Second, no matter what you may have read or been taught, the family of Sargan-type tests are tests of OVER-IDENTIFICATION, not identification. You can "pass" the test and still not achieve valid identification.

Third, passing the tests, useless though they are, in any realistic fashion does not mean failing to reject the null at the .05 or even the .10 level.

Really!

The reason why is that our worry is we might fail to reject a false null. This is type II error. Choosing .05 essentially MAXIMIZES the chances of committing a type II error (minimizes the power of the test). I'd like to see p-values on the order of at least .25 to .30 (or higher).

Since identification is done by assumption, theory becomes super-important. The right way to do this in my view is by recognizing that the equation you seek to estimate is part of a system and the properties of that system will let you know whether identification is achievable or not.

If not, too bad. Estimate a reduced form and be happy.

I pretty much refuse to let my grad students go on the market with an IV in the job market paper. No way, no how. Even the 80 year old deadwoods in the back of the seminar room at your job talk know how to argue about the validity of your instruments. It's one of the easiest ways to lose control of your seminar.

We've had really good luck placing students who used Diff in diff (in diff), propensity score matching, synthetic control, and even regression discontinuity. All of these approaches have their own problems, but they are like little grains of sand compared to the boulder-sized issues in IV.


9 comments:

TheGoodTheBadTheUgly said...

You mean "Sargan" and not "Sargon".

Angus said...

Thanks GBU: I fixed it.

padre said...

Sargon is cooler

padre said...

Sargon is cooler

Chris said...

This problem is only going to get worse. Those of us who did our PhD's at the height of IV hysteria are in the neighborhood of 10yrs out. Meaning many newly tenured professors whose early career was built on IV are entering prime PhD mentoring years.

Julian Schuessler said...

There are tests for instruments when the endogenous X is discrete, see https://en.wikipedia.org/wiki/Instrumental_variable#Non-parametric_analysis

sbik said...

Thank you for absolving me. I have been telling my fellow grad students that this thing doesn't make sense to me. I was wondering why some professors insist their students use it. I have read a lot about it, and my take away is that IV is a neat idea (or method), but rarely useful in practice. Thank you for taking such a strong stance. We, graduate students often want to hear dissenting opinions.

Anonymous said...

Are you saying IV is, objectively speaking, an inferior identification strategy, or are you just saying "economists tend to view IV as an inferior identification strategy?". I'm hoping those two things are not the same to you, or to your readers; else you are all part of why our profession is less like chemistry, more like alchemy.

Angus said...

Anon: IV is an excellent identification strategy when all the conditions are met, but it practice tons of people use it when the conditions are not met but they pretend that they are.