There is a nice little polysci journal called the Journal of Politics. I published one paper there with Mungowitz way back in 1993. I still review for polysci journals and recently got a request from the JOP to review a piece. I said yes, somehow missing in the email the fact that the editors of this journal have completely lost their minds.
Today I got a "gentle reminder" that my review was due on June 9. It was a WTF moment because I had only agreed to review the paper on May 19!! The reminder letter said they had a policy of "reasonable turnaround" so I figured this was a weird typo/screw up. Then I went back and looked at the original request email. And it said that they wanted the review IN THREE WEEKS.
YIKES!!!
Here is the message I sent back to the editors:
"Wow. I can't believe you expect reviews in three weeks. Here is what I can give you. I have skimmed the paper and found it on a first look to be superficial and boring. I have no plans to complete a review in the next week or two. I honestly have to say that your turnaround policy is abusive. I guess I didn't notice the "deadline" TWO WEEKS AGO when I first agreed to do the review. I think you'd better find another reviewer and leave me off your list of potential reviewers in the future. I am still shaking my head in astonishment about the message you just sent me.
Good luck with implementing this policy,
Kevin"
(Somewhere, Don Boudreaux is smiling)
Who in the world are they going to get to do referee reports in 3 weeks, and if they do find people willing to do it, how bad are those reports going to be? Look, referee reports are 3-6 months. That's just how it is. All a 3 week policy is going to do is piss people off.
There is no doubt that long delays in getting feedback from journals is unprofessional and unnecessary. I have a piece with an ex-student that sat for 11 months before we got a review (which was a straightforward R&R) and now the revised version has been sitting for about 8 months. I almost don't remember what the paper is about! But trying to combat long delays by setting an absurdly short deadline is counterproductive and silly.
(UPDATE: Note the ANGUS wrote this, not Mungowitz. And, in the hopes of generating infinite citations, I add this link, that links to us, that links to....)
31 comments:
You know, in those sciences in which results are definitive enough that a finding can be "scooped", the review times found in political science--and I assume in economics--would be considered abusive to contributors. And more highly regarded journals are more demanding of fast turn-around.*
For example, at Science, reviewers are asked to return comments within TWO weeks.
(*: I assume JoP is still more or less the clear number three in the field, with a big gap between the top three and the rest of the pack.)
Yeah, I'm amazed at how long the review process is in economics. I've reviewed about 50 papers in my short career (about 10 per year), and never have I left a review for more than a month. I'm part of a profession--reviewing is a professional obligation. Sorry Angus, this post makes you look lazy.
A couple of options (from a political scientist with papers under review at JoP):
1) Read the email, and if you don't like what they ask, don't review for it.
2) Ask for more time (they usually give it).
Are you really going to read the paper during each day of those 3 months? My guess is no, so all they are asking is for you to put those 2 hours for a review into your queue a little sooner. If you can't do it, then don't do it.
As a political scientist I would have to agree with the others - it is the economists that are screwed up not political science. I have junior econ colleagues who wait over a year for the first round of reviews. Do you have to let the manuscript sit on your desk for 6 months before it is ready to be reviewed? Three weeks is a bit quick, but surely six weeks ought to suffice.
I agree with the above posters. A good review involves reading the paper twice - once quickly and once more carefully. Then, you sit down and pump the review. Total actual work hours for a review usually falls somewhere between 5-10 hours at most. The issue is really finding the 5-10 hours in your work schedule. If you can't do that in a three or six week period, then you need to find a new profession! Or, as they teach teenagers, just say no!
LOL, ok guys. I agree I should have said no immediately. It was my bad not to see the three week deadline in the original email. But you are kidding yourself if you think this is a good policy. Busy academics can't do justice to a complicated paper in that short of time. A good referee report takes time, reflection, deliberation. You are getting half-assed reports on a three week window. I agree that times in econ are too long, I said so in my post. But this is not the answer. Y'all have your system and it's fast and that's fine, but just don't kid yourself that it's leading to quality.
And to the "Anonymous" guy who says I "look lazy": bite me!!
I'll put my work ethic up against yours anytime you'd care to say who you are, pumpkin.
(somewhere, Mike Munger is smiling)
Time allocation problem, no? How many hours does it take to write a reasonable review? How tightly scheduled is your work flow? Why should my career prospects be held hostage to your inability to turn things around within a short period of time.
I don't understand your problem.
I edit a multidisciplinary journal on Latin America, and bug people at two months (no one has complained, or at least not to me). The vast majority complete it by then. I can't really see needing three or more months.
"Busy academics can't do justice to a complicated paper in that short of time."
As I pointed out above--Science gives reviewers two weeks. Now either (a) top-level scientists are not particularly busy, (b) Science does not publish complicated papers, (c) Science suffers from poor reviews that do not do justice to its papers, or (d) busy scientists, when asked by Science, do reviews that do justice to complicated papers in a window even shorter than JoP's three weeks.
Review times in my discipline (History) also run much too long -- 4-5 months is not uncommon, and there's no reason for that. However, it tends to be the higher-rated journals that take longer, and that may reflect that they seek out prestigious reviewers who really do have trouble slotting the necessary 5-10 hours on short notice. Still, as most commentors say, this is more of a time allocation matter: in most cases, the paper for review sits untouched for months, and then is hastily reviewed in the same 2-5 hours it would have gotten right away.
In fairness, paper reviews for journals such as Science are frequently farmed out by PIs to their postdocs; in sole practitioner fields, we don't have that option.
You're just trying to justify your institutionalized laziness.
You haven't succeeded.
If you can't find the time to do a review withing a 4-6 week period (and I'm sure that editors would have no problem with 8) then, yes, you are lazy. And bitching and moaning about this the way you do indicates that you are probably an ass too boot.
In finance journals, turnaround time is 4 weeks. As somebody who referees both finance and economics papers, I can't say that economics papers are noticeably harder to referee than finance papers. And I don't think that the referee reports I've gotten on my own papers from finance journals are noticeably worse.
By the way, regarding Science/Nature asking for reviews in two weeks, this is true. However, it is highly misleading:
(1) They aim to publish only the very most exciting papers. When you get a paper from them to review, you spend 20 minutes skimming it. If it's not so important that you want to make room in your schedule to review it immediately, then you write a brief review saying it's not exciting enough for this journal. This means there's hardly any reviewing burden except for the papers that are exciting enough that you don't mind.
(2) Most Science/Nature papers are very short, just a few pages, and typically fairly straightforward in style. (I.e., describing an experiment and its results, rather than a lengthy or elaborate theoretical argument.) This really makes a huge difference in the difficulty of refereeing.
(3) The review process itself varies between different fields. For example, I know Nature asks authors to suggest some potential referees (with no promise to use those or only those as reviewers, of course); I wouldn't be surprised if Science did as well, but I don't know from personal experience. This is something almost unheard of for journals in my field. I suspect it introduces some bias and game playing into the process, but it does help in making sure that reviewers are close enough in area that they genuinely want to drop everything to read this paper.
(4) Science and Nature are pefectly happy to have false negatives in their reviewing process. They get far more excellent papers than they could ever publish, so they have no problem with rejecting many of them (even for silly reasons). This means they tolerate a somewhat sloppy reviewing process, as long as the sloppiness is in the negative direction. It also means there is extra time pressure on reviewers, since if you don't review the paper fast enough, they may interpret it as a sign of lack of interest and reject the paper for that reason.
In summary, the reviewing process at Science and Nature is an interesting data point, but it tells us very little about how things could or should work at most other journals.
In finance journals, turnaround time is 4 weeks.
Isn't it true that for finance journals, authors pay submission fees and reviewers are paid? I don't know from personal experience, but it's what I've heard from friends (and I just checked two journals on the web, both of which listed submission fees). Maybe this is a more rational system, to make authors compensate others for the burdens imposed on them. However, if this is indeed the way it works, then it's not clear what it tells us about other systems.
Laziness isn't the issue-- but for just the same reason that Angus' "Look, referee reports are 3-6 months. That's just how it is. " is wrong.
An active reviewer is going to review about the same number of manuscripts per year, almost regardless of how long the deadlines are. Say that you're refereeing c. 12 articles per year, just because it's a convenient number. You end up doing each one c. 2 weeks before its deadline.
On a 3-6 month clock, you're currently refereeing a ms you got assigned in January, February, or March.
On a 3-6 *week* clock, you're currently refereeing a ms you got in April or May.
The deadlines come around just as often in either case-- that average frequency is set by the total number of manuscripts you review in a year. "Laziness" would be simply not reviewing many manuscripts. "That's just the way it is" implies that there's something more-than-conventional about the need to take more than 90 days to start a review. But there's not. You can still review a manuscript per month on a 3-week clock. You can still devote just as much time to the reviews, on average, too.
What you can't easily do is suddenly add a bunch of 3-week turnaround times to a schedule that's already crowded with the need to do 3-to-6 month old assignments. Angus' non-lazy schedule presumably already looks like that, and so JOP's request looks like queue-jumping from his perspective.
There's a limit below which you can't just adopt a new disciplinary equilibrium, because reviewing does require some ability to plan ahead and schedule, and no one's going to block out a day in advance to referee the manuscript that on average they expect to receive soon but haven't yet been assigned. At one-week turnaround times, I'd stop accepting assignments. But I think that limit is well below 90 days, and is probably between 3 and 6 weeks.
"[...] all they are asking is for you to put those 2 hours for a review into your queue a little sooner."
You seriously only spend a couple hours putting together your review? Do you even bother to check if the math has blatant errors in it? My colleagues and I put more time than that into class reviews of papers already published in prestigious journals! I realize you get faster with time, but the number of reviews in process also goes up.
Admittedly, there's probably never a great reason for econ reviews to take more than 3 months. Even still, I find it slightly funny that political scientists seem to think they are better at optimizing under constraints than economists.
And Dirty Davey, I wouldn't be a bit surprised to find (c), the reviews are mostly useless. If all papers published in Science are equally excellent, I don't see why we should assume this is a signal that "hard scientists" are better workers (or that their work is more "definitive," for that matter). If anything, I think this signals that the threat of being rejected by a reviewer has more influence on the quality of work than any actual reviewing. In economics, at least, this threat doesn't seem to have the same effect, so there has to be an actual review.
Jacob hit the nail on the head. I can do what I believe to be a quality review in a few days (depends on how long the paper is)...if I drop everything else. In case it matters, my field is computer engineering.
The problem is, I already have weeks' worth of commitments, and I accept reviews accordingly.
Hmm, so this suggests that at least part of the problem is that other journals give reviewers too much time and people's queues are too long as a result!
"They aim to publish only the very most exciting papers. When you get a paper from them to review, you spend 20 minutes skimming it. If it's not so important that you want to make room in your schedule to review it immediately, then you write a brief review saying it's not exciting enough for this journal."
Which has the implication, sadly, that top journals in economics and political science are receiving papers that do NOT sufficiently excite their reviewers... and that (for example) if APSR were to only publish papers so intellectually exciting that people in the field wanted to review them immediately, they would find themselves publishing almost nothing.
How much money should they offer you for you to do it in 3 weeks?
I haven't been a referee for a (strictly) political science journal in more than two years. But I have done many reports for economics journals during that time (of varying levels of perceived quality/prestige) and a few interdisciplinary journals. Three weeks probably is too short of a window due to other commitments (especially, other previous agreements to serve as a referee or, simply, travel). But three to six months? If you really believe that this is what is required to do a thorough, thoughtful, and satisfactory report, then you should get out of the referee business altogether. I don't want to pile on, but this type of behavior/attitude hurts the scientific advancement of our profession.
Dirty Davey writes "Which has the implication, sadly, that top journals in economics and political science are receiving papers that do NOT sufficiently excite their reviewers..."
That isn't necessarily sad. Many of the fields covered by Science and Nature involve things which can be measured by techniques which depend on rapidly advancing technologies (e.g., sequencing/synthesis of DNA, and new semiconductor sensors for fields from chemistry to astronomy). This is a technical reason for a natural kid-in-a-candy-store atmosphere to the work. Imagine what political science might be like if events comparable to the USSR archives opening occurred several times each year.
In journals in technical fields which don't happen to be downstream from such data explosions, e.g. math and algorithms journals, there's plenty of good work, but it doesn't seem to be as surprise-driven. That's not particularly sad, just different.
The current equilibrium in economics generates great uncertainty in assistant professors, as they have to wait a long time to find out if their articles will hit high enough to warrant tenure. This is a common complaint. I would also point out that it forces departments to make tenure decisions based on a much smaller number of publications, which means they make decisions with a noisier signal. This is a cost to the discipline.
Angus,
These comments are fun. You struck a nerve with your post. I figure if someone wants you to work for them AND gives you a short deadline they should be willing to offer the current consulting rate (or at least provide jmcb like incentives). You should have introduced the JOP editors to Mr. Ted DiBiase.
"if APSR were to only publish papers so intellectually exciting that people in the field wanted to review them immediately, they would find themselves publishing almost nothing."
And the downside would be?
To oversimplify, if nobody's excited to review it, nobody's going to be excited to read it, either. So what's the point?
zbicyclicst: "And the downside would be?
To oversimplify, if nobody's excited to review it, nobody's going to be excited to read it, either. So what's the point?"
You don't think a lot of good ideas are sometimes difficult and boring to read?
Quite frankly the impression I get as an Econ graduate student is that the peer review process in Economics in comparison to many other fields is something of a joke.
What's all the more entertaining is that some in the Economics field will defend it. Though I suppose that's hardly a surprise given the over-sized egos and automatic jurisdiction over any area of knowledge that a PhD in Economics gives you...
I'm not a Political Science expert, my elder sister is. From her, I gathered that Political science papers are almost always argumentative in one way or another. Concerning the argument, one should be sure to state a clear thesis at the start of his paper. In introduction, one should not only set out the topic, but also should state clearly what he/she intends to argue with respect to it – the thesis.
I truly agree with Dazy regarding the thesis and paper. You topic is very informative too!!
No. Compare the attitides of the two on Political Compass.In addition read up on how Hitler treated the Jews, then tell me Obama is going to do that in the USA.
I think in econs, it is reasonable to expect a ref's report in 3 months. Apart from delays, what is frustrating is getting a "no" based on a one-liner like "we think your article is more suitable for other journals". Why can't the editor make that decision right away, rather than having the ref do it after a few months? Also, ref reports that are a skimpy one short para, with no solid reasons for rejection. I think journals should insist on a full report with reasons given for rejection. Otherwise, it's easy to reject with a flimsy report, and put in my CV that I was a reviewer for this journal. Some reports I've seen from colleagues obviously show little time was spent by the ref, without much understanding of the article. It's dishonest & unprofessional. Much time was spent by the author (those who got past the editor in the first place), who surely deserves something better.
Post a Comment