Acceptance Rate: 100%
I have been thinking lately about the concept of binary decisions, mainly in the context of paper reviewing. Most of the time, the decision for a paper is binary: Accept a paper or not.
Any binary decision that depends on some explicit or implicit threshold will always be problematic. Whatever is threshold+epsilon gets in, whatever is threshold-espilon is out. An epsilon difference generates significantly different outcomes. To make things worse, the area around the threshold is typically densely populated. No matter where we put the threshold, papers with small differences in quality, even under perfect quality assesment, get very different treatments.
Here is an alternative. Allow all authors to decide whether to publish their papers or not. With one condition. They will also publish together the reviews for the paper. The paper got 3 strong rejects and the authors still want to publish the paper? Fine!
If such a system was in place, then most of the authors would seek to get good reviews instead of trying to pass the threshold and get into the publish-land.
Going a step further? Keep the reviewed versions of the paper together with reviews of earlier versions. Did the authors address the comments? They can have a statement describing that. Later reviewers can take a look and see whether this is the case. This policy would also encourage submissions of only high-quality results.
This can also be matched with the requirement for the reviews to come from at least some high quality reviewers, but I will leave this for another post.
8 comments:
Pano, I wrote this a year ago (intentionally exaggerated, btw); I'm with you! :-) But nobody that I've talked to since then seemed willing to implement anything of the sort (quite a few people, you probably know most of them). So...?
And another point I was trying to make then, but I see is not clear: a couple of centuries ago, all academics in an area could fit in one hall of a "royal society" and discuss face-to-face. Journals and proceedings were a by-product of that discussion, not the ultimate goal. We seem to have forgotten that.
I guess it is frightening to propose such a radical deviation from the current status quo, as the one that you suggest in the post. Perhaps smaller steps would be easier to digest.
Here is an alternative: Papers are submitted. The top ones as "accepted for presentation" (usual process applies). The remaining are also posted with their score percentiles (top-20%, top 30%, etc) and with their reviews. The authors decide whether to include their paper (together with the reviews) with the proceedings, even if it was not accepted for presentation.
Ultimately, as you mention, the goal is to attract attention. A top-5% paper is guaranteed to get more attention compared to a top-40%.
This is easier to implement. Heck, it can even start by listing the reviews for only the accepted papers. Even with "only" 70 papers in a conference, I have a hard time deciding where to allocate my attention.
This system has the side benefit of allowing to judge the accuracy of the reviewing system as well. Are the top-1% papers the ones that have the long-term impact or not? Anecdotes plenty on both sides of the argument. Hard evidence, none.
I like the idea. Will you do something like this for DBrank? Someone has to jump first :-) I *might* try something like this for a future workshop, but would have to warn reviewers way in advance.
Minor quibble:
... authors would seek to get good reviews instead of trying to pass the threshold ...
If by "good" you mean "positive" -- then this is effectively what happens now.
if by "good" you mean "high quality" -- that's a whole other conversation and not clear how authors could optimize for that.
EA
Good: Reviews that will bring the paper to be highly ranked, to attract attention. Being just accepted to a conference will be different to being in the top-10% of the papers, as (presumably) there will be more attention directed to the papers that got unanimously positive reviews.
I see your point. I was thinking that such a system would also discourage half-baked or incremental submissions, but indeed this is not the case. (You need to *force* publication together with the reviews for this.)
A big sticking point in discussions I had was whether the reviews are published eponymously or anonymously. If eponymously (as happens in some journals), it is a scary departure for many. Other arguments include that reviewers may be even less willing to post negative reviews by papers wrtitten by "authority" figures. If anonymously, there is still no way to score reviewers and build a "reviewer reputation" rank and there is little motivation to write high-quality reviews (i.e., not much difference than the current situation). I really have no good answer.
On the more radical suggestion of completely open Digg-style ranking there are many issues about reviewer incentives (most people will just not bother if they are not obligated to review). But, you're right, all these are far too great departures and perhaps too scary (and too contentious), at least for now. An interesting suggestion someone gave me on Digg-style ranking, though, was to do it not in the submitted papers, but in online proceedings, in a *separate* process, with participation by conference attendees initially and subsequently by everyone. There are already systems that sort of do this, but not widely adopted.
I like the idea. To some extent we have it already: it's called the web. Anyone can self publish anything. Then others can leave comments and reviews. Still others can serve as editors, recommending good content.
Still today, self-published articles (one example of which are blog posts) aren't viewed as credible in academic circles.
I like the idea. I've always felt like the accept/reject decisions are largely luck!
I'd be curious for journals and conferences to send out "control" papers that multiple reviewer-groups would read and review without knowing that they were the ones being evaluated. Such a control paper would be a good guage to show how much luck is involved in getting over threshold+epsilon!
Post a Comment