referee Hall of Fame/Shame

At the Pontiff, we are big fans of Science 2.0 in all its forms. But even within the traditional journal system, there are ways to improve the peer review system. One tragedy of peer review is how little credit the referees get for doing good work, and how there’s nothing to trouble them if they don’t but their own guilty consciences and a bunch of emails from an editor.
One approach I recently came across is from an economics journal. They publish a list of their associate editors, together with average turnaround time for manuscripts.

Editorial Board Member Manuscripts
Name Affiliation Reviewed Avg Days
1 Bessembinder, Hank

University of Utah

8

14

2

DeAngelo, Harry

University of Southern California

4

18

3

Dittmar, Amy

University of Michigan

4

25

4

Duffie, Darrell

Stanford University

2

12

5

Fama, Eugene

University of Chicago

6

2

etc.

This is a nice start, but surely we could do more. When I teach, I get student evaluations. What if the authors of the papers I refereed also gave me 1-5 stars as a reviewer? Before you mention the obvious problem, the way to average the ratings would be by outcome: all the “reject outright” ratings are averaged together, all the “revise and resubmit” ratings are averaged together, etc., before these are all combined into a final score. That way, you couldn’t get high ratings just by always accepting everything.
Clearly this proposal still needs more work. Any ideas?

4 Replies to “referee Hall of Fame/Shame”

  1. I’d like to think that I would still give few stars to a reviewer that accepts the paper but doesn’t have anything interesting to say or clearly didn’t understand some aspects.

Leave a Reply

Your email address will not be published. Required fields are marked *