Public Paper Reviews
I am now trying to review a journal paper. Unfortunately, the paper is completely unreadable, being a long list of definitions, "lemmas," and "theorems" (the quotes are intentional). Even more unfortunate is the fact that the author does not give any background but rather cites some recently published paper of his, which contain all the necessary background.
Trying to understand the paper, I attempted to read the published paper. Well, no luck! The paper was also horribly written. I started wondering who in his right mind decided to accept this paper to an ACM journal, and what the reviewers were saying. I simply cannot believe that anyone in his right mind would actually read such papers, and even try to write any meaningful review afterwards. Most probably the reviewers gave up and agreed after many revisions to allow the paper to be published, hoping that they will never have to read the paper again. Or may be I am wrong and the paper is indeed a hidden gem?
So, how can we avoid such cases? Here is my not so original suggestion: Publish the (anonymous) reviews together with each paper!
I cannot see anything negative with that. It will make everyone happier. People that write high-quality reviews would not mind seeing their names being published together with the reviews. Other reviewers will see what is a high-quality review and hopefully will try to imitate the style.
In fact, the practice of publishing a commentary for each paper is not new. I have seen many papers in statistics being published with eponymous commentary. Often, reading the reviews is more interesting than reading the paper.
Furthermore, the reviews will offer a quick overview of the contributions and shortcomings of the paper. It will also allow the reader to understand what lead to the acceptance of the paper. Was it a new idea? An excellent experimental evaluation? Or just the reviewers could not even read the paper and just gave up, giving a lukewarm "accept"?
14 comments:
Are you suggesting that the published reviews retain or lose their anonymity? I think the latter would result in bland, inoffensive reviews--or in fewer people being willing to participate in the review process. There's a lot more downside for offending someone than upside for gaining a reputation as a good reviewer.
I think it is important to keep anonymity as an option.
But reviewers that write good reviews (good does not imply positive) should have the option of keeping their name attached to the reviews.
Also, keeping the reviews for published papers should ensure that the reviews are rather positive and at least highlight the good points of the paper.
How long has reviewer anonymity been standard practice anyway? I recently reviewed some historical stuff on Darwinism and the debates that followed. There were many public reviews of various people's papers/books.
I agree with the general idea, though the reviewers should have to seperate their comments in public versus private (editor's eyes only) sections.
Here are three basic issues with the idea:
1) What about lazy people who read only the abstract and the reviews? All it takes is one cranky reviewer to negatively scar the impact of the paper.
2) What if a reviewer was just wrong on one or more points yet gave a negative review? This one is a corollary to #1
3) Currently the voting policy is IN or OUT. Would publishing the detailed numerical voting by category institute a defacto ranking of papers within a journal?
To be a bit contrary here... one rationale for NOT publishing reviews is to make the reader actually consume the material and not look to use the reviews as a crutch or 'Cliff Notes' assessment.
Neal, good points.
If I remember correctly, anonymity started around 1920's or 1930's, and proliferated after WWII. Before that, it was the editor that made the decision, potentially with the help of the associate editors. (I think that Einstein found it insulting when an editor gave one of his papers to an anonymous reviewer.)
I think #1 and #2 are less likely for accepted papers.
For #1, I kind of doubt that a reviewer will accept a paper by reading the paper very quickly.
For #2, for incorrect points of a reviewer, I think that publishing the author's response will take care of this. In fact, reading the debate about an ambiguous point may be actually be more beneficial for a reader. Myself, for some papers that are discussed extensively in PC meetings, I keep the discussion archived, together with a copy of the paper. It helps me quickly understand the strengths and weaknesses of a paper, going beyond my own reading.
For #3, I think this is a good thing. No reason not to allow additional reviews on a paper. There is already such an effort from ACM (www.reviews.com) but unless the discussion is "seeded" by some trusted reviewers, nobody wants to be "that guy" that was the first to point out flaws to a great paper, or someone that liked a stupid paper.
On a related note, SIGMOD Record used to have a column on "influential papers", maintained by Ken Ross that published reviews of classic papers that influenced significantly the reviewer.
Clarification note: The 3 points above were directed at problems post publication for downstream readers of the paper. Not at reviewers being lazy.
Isn't this what 'letters' are for? The example I love of a nice exchange is below, it's classic, funny, informative and an example of where public comments could go badly wrong.
http://arxiv.org/abs/cond-mat/0108530
http://arxiv.org/abs/cond-mat/0202383
http://arxiv.org/abs/cond-mat/0203275
http://arxiv.org/abs/cond-mat/0303261
While Goodman's comments to the original are infomative, his behavior was impolite. My litmus tests of the original paper was "Is this interesting?" and "Will I tell others about it?", the answer to both is yes.
Why don't more Journal's have a letters area? I suppose that is what blogs are for these days?
Whether it is impolite or not it can be judged by the readers. At the very least, Goodman puts his name under his opinion, allowing others to judge the correctness of his arguments. And at the end of the day, would you prefer to have read just the original paper, or did you like the exchange of opinions that followed?
The "Letters" section is nice to have, but from my experience it is dominated by negative opinions, not positive ones. Furthermore, not all papers will generate a reaction from the readers. And we still need to have a way to seed the discussion.
The paper(s) were more memorable to me precisely because of the exchange! In general I support the public disclosure of reviews... though I want some editorial control to prevent the more negative academic heckling potential. It's not supposed to be the blogosphere! Naive?
Who wants to read "the figures could be clearer, particularly Figure 4 in Section 3.1 ..."? And what do you do with the reviews of papers that get rejected?
My suggestion to improve peer reviewing is twofold. First, charge submitters a reviewing fee. This discourages poor-quality submissions. And second, pay peer reviewers for their reviewing.
The only example I know of public reviews is alt.chi 2009.
http://www.chi2009.org/altchisystem
See the "accepted papers" section.
William, the idea of paying per submission has been around for a while. I wrote about this in the past http://behind-the-enemy-lines.blogspot.com/2008/04/how-much-paper-submission-costs.html and I there was an article at CACM this January http://cacm.acm.org/magazines/2009/1/15664-viewpoint-scaling-the-academic-publication-process-to-internet-scale/abstract
It would be interesting to see real monetary prices for submission. I assume that most reviewers do not really enjoy writing reviews, so a consulting rate would apply (~$200/hr). For 3 hrs to review a paper (conservative estimate), and 3 reviewers per paper, this is ~$2000 per submission. Yes, it would be nice to see what would happen when people realize how much each of the submissions costs... ;-)
Sergio, this is a very nice example. It is actually very helpful to have access to the reviews. Granted, it does not seem to be a high-stakes conference, but for a first experiment it looks very good.
Panos,
Hi! Thanks for the pointers. I obviously need to leave academia and take up consulting...
William
Panos,
Hi again! I'm reading the rationale for VLDB's move from conference-only to journal+conference format:
http://vision.jdmr.org/
They mention in this that in the journal format, each published paper will be accompanied by a paragraph from one of the reviewers that summarises the paper's contributions and importance. (Search for "champion reviewer".)
William
Post a Comment