Positive or Negative? The Economist, Prediction Markets, and Mechanical Turk
A few days back, the Economist published a short story about prediction markets, saying that the concept has not taken off as many people were expecting. Chris Masse from MidasOracle got a little obsessed about the piece, publishing many blog posting about the article, and even making calls to everyone to post answers and discussion in their blog. Everyone who just read the blog post would think that the Economist article was pretty much tearing down the concept of prediction markets, their practicality, and their usefulness.
Michael Giberson actually responded saying that the article was pretty much a run-of-the mill article, and there was nothing to respond to and that the article was not even negative. Chris listed a long list of argument why the article was negative.
OK, so is it positive or negative? Well, let the crowd decide. I posted the article on Mechanical Turk, asking 100 Turkers to read the article and rate it in a scale from 0 (most negative) to 10 (most positive) about the sentiment of the article towards prediction markets.
Here are the results:
Well, the average was a 5.8/10, meaning that the average detected sentiment was pretty much neutral with some hints on positivity.
The lesson: never let your own judgment drive your decisions. Measure and analyze the data!
Sorry Chris! Too much drama for nothing!
10 comments:
20% of the people polled graded the article 8/10.
They should be brought into a psychiatric hospital. There are many mentally ill people out there. :-D
Joke aside, I acknowledge the results, and I thank you for your time.
Wouldn't you find a bell curve in this kind of situation? If you ask reasonable people about grading something, is that normal to have grades all over the place?
I would like to take 10 PhD people, and ask them the same question. Maybe the question should be finessed, too. I am wondering...
Were they jusy random users of Mechanical Turk, or were they users with whom you were familiar?
Is the fact that people use Mechanical Turk, not likley to bias their decision. Is the fact that it was you posting the survey, not lkely to bias the results?
Is this the collective wisdom of crowds at work again?
Can yoiu honesly say that you have analysed this data?
What is important is the opinion of the CEOs and other business leaders, since EPM software and consultancy is a business-to-business industry. Business people should be polled about it, not consumers hired for cheap on the Web.
Chris: It just seems that people have different opinions, so the result looks more like a "mixture model" rather than a single Gaussian. Take the result for what it is: when someone reads the article it does not feel like a disaster.
Partido: I do not pick users, they self-select. Furthermore, they do not know my name when I post the task, as I use a pseudonym for the tasks that I post. I do not see much of a reason for bias in this "survey". The fact that Mechanical Turk users are good at NLP tasks is actually examined by others as well (see the Dolores Lab blog), so I am confident that the results are reliable. And yes, it is the collective wisdom of the crowds at work. And no, I did not analyze the data. I just reported them.
"It just seems that people have different opinions,"
When the results are all over the place, is the "average" a good metrics?
What's the average between the height of the tallest basketball player and the height of a dwarf?
Chris: Yes, the average may not be the single best measure, but the chart shows the distribution, too, so there is more information.
On the self-selection of Turks, I can imagine that participants differ from, say, the population of all people in significant ways, but the participants are probably are not as different from the population that reads The Economist.
Might be a source of concern, but still data from 100 readers is more information that the interpretations of a small number of bloggers.
Report the quantiles then. That's why I give the histogram and not only the summary statistics.
"Might be a source of concern, but still data from 100 readers is more information that the interpretations of a small number of bloggers."
Amen, Michael
"the participants are probably are not as different from the population that reads The Economist."
Maybe, but what counts in the end is the opinion of the CEOs reading The Economist. Will they want to pay $$$ to implement such a tool described in The Economist?
EPM software industry = a business-to-business industry.
What matters is the opinions of the business people... and the opinion leaders writing in newspapers, magazines, and big blogs.
I would welcome a poll of business people and opinion leaders who have read the article by The Economist.
That said, thanks to Panos for his time working on the topic, and the great comments on this blog and elsewhere.
Post a Comment