Maybe it’s not a war, but it is at least a skirmish.
The first shot was fired by a pseudonymous user named gray, who apparently has never scited any papers before and just arrived to bash an author of this paper for using a recommendation engine to… cue the dramatic music… recommend his own paper!
In an effort to stem this and future carnage, I’m taking to the quantum pontiff bully pulpit. This is probably better suited for the SciRate blog, but Dave didn’t give me the keys to that one.
Since it wasn’t obvious to everyone: SciRate is not a place for trolls to incite flame wars. Use the comments section of this post if you want to do that. (Kidding.) Comments on SciRate should have reasonable scientific merit, such as (at minimum) recommending a paper that was overlooked in the references, or (better) posting questions, clarifications, additional insights, etc. As an example, look at some of the excellent substantive comments left by prolific scirater Matt Hastings, or this discussion.
Nor is SciRate the place for insipid dull self-promotional comments and/or gibberish.
Now to make things fun, let’s have a debate in the comments section about the relative merits of introducing comment moderation on SciRate. Who is for it, who is against it, and what are the pros and cons? And who volunteers to do the moderating?
As for “gray” or any other troll out there: if you want to atone for your sins, my quantum confessional booth is always open.
O brave new quantum world!
I’m a big fan of TCS StackExchange and MathOverflow. Particularly commendable (IMHO) are the checks-and-balances that both sites have installed, with an view toward preventing what a recent Slashdot article calls Gamification — Valid Term or Marketing-Speak?
Upon visiting the SciRate site (for the first time) the role of gamification wasn’t clear … certainly many of the recent SciRate comments were wholly concerned with the ins-and-outs of “gaming” the ratings.
One wonders, how do institutions like the AMS’ MathSciNet (which is a wholly admirable institution IMHO) handle gamification-related issues? Are there any STEM disciplines that have deliberately and successfully embraced gamification? Would fields like CSE/CT/QM/QIT benefit from increased gamification? Are on-line prediction markets examples of successful gamification? What about Vegas sports betting?
And finally … are the preceding questions good for starting the requested flame-war? 🙂
I know someone who Scites his/hers own papers to increase their visibility, and unscite them when enough people voted for them. This way (s)he does not look so presumptuous.
To answer the question, since the stn ratio is quite high on Scirate, I think it is not worth censoring comments…
Dave, somehow I had the (totally mistaken) impression that SciRate was an iPhone app. A little more seriously, I’ve seen the results when medical centers and academic departments play “the rating game” … it ain’t pretty.
Somehow, SciRate has to be about more than ratings. Which is the same as saying, that science somehow has to be about more than relative prestige … and whatever that “more” might be, SciRate has to focus/speed/improve/teach/summarize it. We will say, “Googlify” it. 🙂
Damn should have read this post before flaming the war. Doh.
More seriously who wants to help rewrite scirate from scratch? Personally scirate was my way of 1) learning about ajax, SQL, PHP, and python and 2) having you all do work for me (okay…everyone wins in crowd sourcing). But I think all of you could come up with a much better way of making a site like this!
@johns I can’t believe you’ve never visited scirate until I left UW 🙂
I think scites are like any other measure of academic quality in that they’re likely to be extremely accurate until they become important for anything, and then people start trying to influence them.
One way around this would be if scites were like the google “+1” and your view of scites could be shaped by your social network in a pagerank-like fashion. If only some google engineer had 20% of their time to devote to this important task…
@Dave, I can help.
I would not advise moderating comments. It would be manageable to do it now, but it is not scalable and the goal of SciRate (if indeed it has one) should be to scale to a size where it becomes a truly useful metric. It would be good to do a rethink/rewrite of SciRate. I am not sure that a pure ranking site is the best model. Sites like Digg and Reddit are no longer where the action is. I agree with aram that there needs to be some way to bring your social graph into it. Since a very large number of qinfo people seem to have joined Google+, perhaps it would be best to wait for the API to come out and think about building it on top of that. Neither Facebook or Twitter are ideal networks for this purpose, and it would be a mistake to try and make SciRate into a social network in its own right. I don’t need another site where I have to go in and decide who to follow.
I’m not sure that counting self-scitations is of much use, but I don’t really care that much about it since I scan all the postings myself anyway.
The comments seem like a useful (although hit and miss) way to communicate with the referees and program committees that an article contains errors or lacks proper citations. I’m not sure that it’s worth anyone’s time to try to referee the resulting flame wars, although it’s satisfying to see an occasional drunk get tossed out of the bar.
I’d like to upvote everything Matt Leifer said (JOKE). Upvotes work well for ranking pictures of cats and smartarse comments, and also for burying rude or unconstructive contributions. They also seem to work extremely well on Stack Overflow type sites for ranking short(ish) answers to technical questions. It’s less clear to me that they are useful for ranking papers. It usually takes me a couple of days to properly absorb even a 4 page paper, and yet some of the most prolific sciters vote on 3 or 4 papers in a day. I’m rather skeptical that such people are reading all the papers they scite – it’s likely they’re just upvoting names that they recognize. The utility of this information is not clear to me, I already know who the “big names” in the field are and I can easily find their papers by skimming the daily arxiv postings.
Far more useful would be if the ranking information were collected passively, like the scrobbling done by last.fm. I know I can rely on that data because it’s based on what people are actually listening to, rather that what they claim to like (but perhaps never play!) In the case of scientific papers, it would be really useful to know how often a paper has been downloaded, viewed on screen, or even how often it’s been printed out. Combined with more nuanced information about one’s social graph, this could be the basis of a useful recommendation engine.
I scite every paper I might imagine ever wanting to read later. Other people save their scites for papers they consider good, or maybe they’re just more realistic about what they’ll get around to reading. But one issue with interpreting scites is that people who use them have generally different ideas about what they mean.
I suppose it’s fairer to say, then, that scites are really mostly telling us something about people’s prior expectations of whether they’ll find a paper interesting or not. I think this is still useful, and probably more useful than I implied above: Scirate tells me something about the prior of the whole community (at least that part of it that uses scirate), which won’t overlap precisely with my own.
Perhaps Sicrate 2.0 could be more ambitious than this, though. If people are using it as their main point of access for the arxiv, one could harvest info about which abstracts are being expanded, and which papers are actually being downloaded. Maybe have the front page not display the most recent papers, but rather the “most interesting” from the preceding few days (according to whatever algorithm gets settled on). Then one could do things like weighting a scite on a 3 day old paper more heavily than a scite that comes in 30 seconds after the paper first gets posted (because it’s more likely to be based on a critical appraisal of the content). Likewise, a comment on a paper should count towards it’s score. People who regularly make substantive comments on the site could have their scites weighted more highly than others (though I can think of arguments against this last suggestion, too).
More suggestions: we might want to think about exactly how new papers are presented, especially in light of Paul Ginspargs findings on the “positional effect”: http://arxiv.org/abs/0907.4740 and http://arxiv.org/abs/1010.2757 . It’s shockingly strong: “hep-th and hep-ph articles announced in position 1 had median numbers of citations 50% and 100% larger than for positions 5–15.”
There might be a similar effect on Scirate, which then snowballs as the top papers get scited even more. It might be interesting to try and subvert this bias (and other obvious ones), e.g. by randomizing the order in which brand new papers appear (but then ordering “few day old” papers according to merit), hiding the absolute score that a paper has, or hiding the authors names (but of course allowing them to be revealed with a mouse click).