Mechanical Turk: The Demographics
One of the common misbeliefs about Mechanical Turk is that it is a virtual sweatshop, essentially taking advantage of poor people in third world countries that are doing tedious tasks for pennies. Therefore, many people are afraid of outsourcing research tasks on Mechanical Turk, being afraid that the results will be either of very poor quality, or they will not be representative of the actual U.S. population.
Those that read the previous, qualitative survey about Mechanical Turk would have realized that the profile of the typical Turker is not of a person that completes tasks for a living in a developing country. Instead, Mechanical Turk tends to be often a replacement for TV, or simple something to spend some free time and get some spare cash in reward.
The next survey that I conducted focused more on the demographics of the Turkers. Are they uneducated, unemployed people with no income? Well, as you will see below the Turkers are a pretty representative sample of the online population, perhaps with a slight bias towards females and towards young participants. Let's see the results!
First, I would start with the country breakdown.
| United States | 76.25% |
| India | 8.03% |
| United Kingdom | 3.34% |
| Canada | 2.34% |
The clear result is that most of the participants are coming from the US and not from a third world country, despite the common misconception. This is due to the fact that in order to get paid, someone has to have a US bank account, or be willing to be paid using Amazon gift certificates.
Then, the gender breakdown:
As you can see, there are slightly more females that males. I do not have a definite reason yet, but I get a feeling that females are less inclined to "waste time" and find that if they can exploit their spare time to get a little bit of income, then they would do it.Next, the age distribution:
Not surprisingly, many young people participate on Mechanical Turk, mainly as a way to get some extra cash and to be able to drive their car, get some items from Amazon and so on. (The Mechanical Turk payment can be either deposited in a US bank, or be given as an Amazon gift certificate.)And what about education?
Turkers are a pretty representative sample. Most of them have a college education, and some of them even have PhDs! In fact, the distribution seems pretty similar to the distribution for the overall US population.Similarly, the income distribution also follows closely the income distribution in the US:
Finally, why people participate in Mechanical Turk? From the qualitative survey, you could see that most of the participants mention money, one way or another. However, very few participate only for the money. (See also the detailed responses.) Here is the breakdown of the responders when they had to choose (not exclusively) between the choices "for money," "for fun," and "for killing time":
I hope that the results above shed some light. I have to thank my student Beibei for preparing the running the survey for me. The next steps now are to present the results of the qualitative survey in a coded/tabulated manner, and to give more details about the different tasks that we had run on Mechanical Turk and the lessons that we learned.If you have any more questions that you would like to see answered, let me know!
Update: See also the responses about the amount of money that Turkers earn, and a more detailed breakdown of the motivation of Turkers.
10 comments:
Very interesting. This however doesn't answer how well performed the tasks actually are. Demographics is one thing, how about a survey of the Requesters; are they really satisfied with the results? How many rejects in average in the different categories, which kinds of tasks are most suitable for AMT? etc.
David,
I am preparing a more extensive discussion about the different tasks that we run on MTurk, what worked, what did not, what are the best practices and so on.
If you want a quick answer: You need small tasks that have preferable a multiple choice outcome. You accept tasks only when more than 1 people agree on the result (the exact number depends on the desired confidence). If you have a free form answer, then it is better to have an extra, "validation" task where other MTurkers votes on whether a particular answer is correct or not.
OK, more details in a future posting :-)
Very interesting!
I know of an internal survey at a Silicon Valley company that pretty much has the same results.
I suppose you created HITs and just asked a subset of "random" turkers (everybody who happened to do it) about their age, gender etc.? Is this correct? On how many turkers is the survey based?
BTW: I'd expect that you would get slightly different results according to the time you put the HITs online.
Thanks!
Michael
We asked 300 Turkers to complete the survey, and it took 2-3 days. I have not performed any cross-validation to see if time of the day, or day of the week play an important role in terms of demographics.
This is an interesting report. I would be curious what the questionnaire looked like that turkers answered for you to collect these demographics. Still, I think the data would be biased to users who chose to respond--not necessarily representative of the actually mturk population.
To David Andersson's comment about how well the tasks are performed: that greatly varies. My colleagues published "Crowsourcing User Studies with Mechanical Turk," suggesting ways you can formulate questions to get good, accurate responses (including validation tasks as Panos suggested) http://moourl.com/ppolz
Similarly, I have run several studies on Mechanical Turk now with excellent results. My surveys were not multiple choice questionnaires--question formulation is important, but each survey question was specific, simple, and "flowed" from the previous question. It was even pretty long and I still got great data. A short post about this is here: http://brynnevans.com/blog/2008/07/09/using-mechanical-turk-for-research/
Panos, I briefly looked over your methods in the Faceted Hierarchies paper, and thought it looked great! I'd be interested in talking more about your experience using Mechanical Turk. I thought you weren't supposed to ask turkers identifying information like demographics? Or perhaps demographics aren't identifying on their own?
The Augmented Social Cognition group at PARC has a parallel conversation about the general topic of demographics on Mechanical Turk, based on your original post on this blog: http://asc-parc.blogspot.com/2008/07/mechanical-turk-demographics.html
Thought it might be of some interest to merge the conversations.
Taking jobs on Turk for peanuts is pretty much a waste of time, so I think this indicates that females are more inclined to 'waste time', not less. Alternatively, you can frame it that men value their time a little bit more (perhaps because society tends to unfairly value that time a little more highly?). What you can't reasonably conclude is that people who participate in Turk are less OK with wasting time than those who do not, because you have no idea what people who don't do Turk are doing alternatively - maybe they are making *more* money. In fact, they probably are! Because it's pretty *hard* to make less for the same effort.
@Anonymous: You simply don't get it, do you?
People seem to participate while watching TV, or while *not* working in the office. No single Turker mentioned that they use it instead of some other job; all of them use MTurk during their spare time, instead of websurfing, or playing online games...
Females need extra money because they make 66cents on every dollar that a man makes.
Hi Panos,
I've just run some research on a small sample (n=100). I got demographic data from it and it agrees with your findings.
Post a Comment