Soliciting Reviews on Mechanical Turk
A few days ago, Mechanical Turk was featured in a not-so-flattering story: Mike Baynard, a Business Development Representative working for Belkin has posted HITs on Mechanical Turk asking Turkers to write 5 star reviews on Amazon, for a set of Belkin products that were getting mainly negative reviews. Furthermore, Mike Baynard was asking Turkers, after posting the 5/5 review, to vote as "not helpful" the negative reviews that appeared on Amazon. The story was picked by major tech sites (Gizmodo, Slashdot) and, fortunately, users put the blame on Belkin and not on Mechanical Turk.
However, this got me thinking. How often is Mechanical Turk used for such sort of activities? Fortunately, a few weeks back I set up a crawler that visits Mechanical Turk periodically and keeps track of the tasks posted there.
So, in my first use of this MTurk archive, I digged in and tried to find review-related HITs. I discovered about 100 HIT groups posted over the last couple of weeks. Except for three HITs posted by Michael Bayard, the rest did not seem to explicitly solicit positive reviews. In fact, most of the requests seem to be legitimate and in my opinion, ethical. Yes, in an ideal world all websites would get millions of users submitting reviews and generating network effects, but sometimes it is better to pay and have a review than not having a review at all!
(The reviews are listed in a table, which is contained within an IFRAME; this means that you will need to visit the blog page to see it as RSS readers typically do not render IFRAMEs)
Now, the question is: Are these solicited/paid reviews better or worse than the reviews posted by users without any financial incentive? I expect to have some results in that front rather soon.
6 comments:
Thanks Panos. I agree that most of the blame has been directed at Belkin, and rightly so, but I also believe Amazon deserves to be dinged a bit more than they have been for this. Effectively they are profiting from spam. And these types of activities appear to be violations of mturk's terms of service, so IMHO I believe Amazon should do a better job of policing these activities.
I don't see much distinction between mturk in this regard and buyblogcomments.com, for example, a service where you can hire people to spam blogs:
http://mblog.lib.umich.edu/~jmm/archives/2007/09/incentive_wars.html
http://blog.oddhead.com/2009/01/14/intelligent-blog-spam/
Perhaps Amazon should first pass the submitted HITs through MTurk itself to make sure that they follow the terms and conditions of the service. :-)
Of course, as HIT creators accumulate reputation and submit X number of good HITs, then it would be possible to let them post without any filtering.
StreetAdvisor.com acknowledges that it uses paid authors via MTurk for some reviews in the US. We do this to establish content in geographical areas to demonstrate the purpose and function of the site, which is an effective way to encourage genuine user participation.
We make no direction about whether reviews must be positive or negative. We ask that all reviews contain honest comments about personal experiences on streets and in cities, such as access to transport and stores, availability of schools and community services, and quality of life and interactions with neighbors.
Hi Panos. Really enjoyed your past post and thanks for your insight here about soliciting reviews on mturk. Clearly the practice by the Belkin employee was not very smart. At web2review.com, we have employed Turkers to add basic profile information for websites, such as tags, or whether the site is free or paid (as noted above). We have not paid any turkers to write reviews and don't believe this is best practice. I don't believe its fair to associate us in this blog post just becuase our name is web2review and we show up on mturk. I don't think our stuff is good evidence for a credible blog post. Feel free to check out any of our HITs online.
@web2review: I removed your entries from the table, as indeed they were not soliciting reviews.
I would like to say, though, that I do not consider unethical the practice of paying for reviews. If it is necessary to pay for gathering reviews, let it be it. Newspapers and magazines are also paying their critics. I wouldn't say that anyone considers such reviews not trustworthy!
Hi Panos, I appreciate your follow up thoughts and for removing the unassociated entries. Although our web2review.com site is open for sneak preview, we havn't yet officially launched... so we didn't want to get off to a bad start. Thanks!
Post a Comment