Tuesday, April 5, 2011

An ingenious application of crowdsourcing: Fix reviews' grammar, improve sales

I have been doing research on the economic impact of product reviews for a while. One thing that we have noticed is that the quality of the reviews can have an impact on product sales, independently of the polarity of the review.

High-quality reviews improve product sales

A well-written review tends to inspire confidence about the product, even if the review is negative. Typically, such reviews are perceived as objective and thorough. If we have a high-quality negative review this may serve as a guarantee that the negative aspects of the product are not that bad after all. For example, a negative review, such as "horrible battery life... in my tests battery lasts barely longer than 24 hours," may be perceived as positive by other customers that consider a 24-hour battery life to be more than sufficient.

In our recent (award-winning) WWW2011 paper "Towards a Theory Model for Product Search" (with Beibei Li and Anindya Ghose), we noticed that demand for a hotel increases if the reviews on TripAdvisor and Travelocity are well-written, without spelling errors; this holds no matter if the review is positive or negative. In our TKDE paper "Estimating the Helpfulness and Economic Impact of Product Reviews: Mining Text and Reviewer Characteristics"  (with Anindya Ghose), we observed similar trends for products sold and reviewed on Amazon.com.

And what can we do knowing this?

Being in a business school, these findings were considered informative but not deeply interesting. Do not forget, the focus of researchers in business schools is centered on causality and on policy-making. Yes, we now know that it is important for the reviews to be well-written and informative, if we want the product to sell well. But if we cannot do anything about this, it is not deeply interesting. It is almost like knowing that during the cold months the demand for summer resorts drops!

But here comes the twist...

The crowdsourcing solution

Last week, over drinks during the WWW conference, I learned about a fascinating application of crowdsourcing that attacked exactly this issue.

An online retailer noticed that, indeed, products with high-quality reviews are selling well. So, they decided to take action. The retailer used Amazon Mechanical Turk to improve the quality of the reviews posted on its own website. Using the Find-Fix-Verify pattern, the retailed used Mechanical Turk to examine millions of product reviews. (Here are the archived versions of the HITs: Find, Fix, Verify. And if you have not figured out the firm name by now, the retailer is Zappos.) For the reviews with mistakes, they fixed the spelling and grammar errors! Thus they effectively improved the quality of the reviews on their website. And, correspondingly, they improved the demand for their products.


For the curious readers, Zappos has been doing this at least since April of 2009, which means that they were doing it even before being bought by Amazon.

While I do not know the exact revenue improvement, I was told that it was substantial. Given that Zappos spent at least 10 cents per review, and that they examined approximately 5 million reviews, this is an expense of a few hundred thousand dollars. (My archive on MTurk-Tracker kind of confirms these numbers.) So, the expected revenue improvement should have been at least a few million dollars for this exercise to make sense.

Ethical? Notice that they are not fixing the polarity or the content of the reviews. They just change the language to be correct and error-free. I can see the counter-argument that the writing style allows us to judge if the review is serious or not. So, artificially improving the writing style may be considered as interference with the perceived objectivity of the user-generated reviews. I still consider it fine to change the grammar, from the ethics point of view.

But is it ingenious? A resounding yes! It is one of these solutions that is sitting in front of you but you just cannot see it. And this is what makes it ingenious.