At the Pontiff, we are big fans of Science 2.0 in all its forms. But even within the traditional journal system, there are ways to improve the peer review system. One tragedy of peer review is how little credit the referees get for doing good work, and how there’s nothing to trouble them if they don’t but their own guilty consciences and a bunch of emails from an editor.
One approach I recently came across is from an economics journal. They publish a list of their associate editors, together with average turnaround time for manuscripts.
Editorial Board Member Manuscripts Name Affiliation Reviewed Avg Days 1 Bessembinder, Hank University of Utah 8 14 2 DeAngelo, Harry University of Southern California 4 18 3 Dittmar, Amy University of Michigan 4 25 4 Duffie, Darrell Stanford University 2 12 5 Fama, Eugene University of Chicago 6 2
etc.
This is a nice start, but surely we could do more. When I teach, I get student evaluations. What if the authors of the papers I refereed also gave me 1-5 stars as a reviewer? Before you mention the obvious problem, the way to average the ratings would be by outcome: all the “reject outright” ratings are averaged together, all the “revise and resubmit” ratings are averaged together, etc., before these are all combined into a final score. That way, you couldn’t get high ratings just by always accepting everything.
Clearly this proposal still needs more work. Any ideas?
Amused that Fama is apparently doing his bit to prove that the market for ideas is at least fast, if not efficient.
To reduce correlation with outcome, one could also ask authors to simply rank the reviews.
I’d like to think that I would still give few stars to a reviewer that accepts the paper but doesn’t have anything interesting to say or clearly didn’t understand some aspects.
The question is how many stars you would give to referees who rejected the paper.
I guess the editor could also award the stars.