Mechanical Turk: Profitable or Not?
I was chatting with a couple of students of mine, who are trying to build a startup using Mechanical Turk. So, they asked me what is the value of the tasks that are being posted every day on Mechanical Turk.
Since I have the archive of the tasks posted on MTurk for the last couple of months, this was an easy question to answer. A simple query on the database, find the new HITs posted every hour, group by day, and here is the plot:
Long story short, the average value of HITs posted in any day is around \$2000. I have not analyzed the distribution of the values, but it seems to be (not surprisingly) a power-law or a lognormal.
The 2K/day value means that the average revenue per day for Amazon is around 200 per day (10% of the requester's payment), or 6K/month. This hardly covers the expense of dedicating a developer to the service!
It seems that Mechanical Turk is not generating any significant revenue for Amazon. It is also unlikely that it generates any profit. And we know these days what happens to products that are not generating profits...
Thankfully Amazon uses Mechanical Turk for its own purposes, so there are second-degree benefits for Amazon to keep the service around. I truly hope though that the service will attract more customers soon.
Update: I should also clarify that my figures are slight underestimates of the actual figures: I can only "see" Mechanical Turk through the eyes of an average worker. So I cannot see if a requester asks multiple people to complete the same HIT and, sometimes, I cannot observe the details for HITs for which I have not passed the qualification test. I still think that the numbers will be at the same order of magnitude.
Update 2: If you want more details see my previous post on the dynamics of Mechanical Turk and the chart at http://hyperion.stern.nyu.edu/mturk/ that shows the available HITs and rewards at any given time.
19 comments:
Panos:
Is the $200/day figure the amount paid from contractors to the Mechanical Turkers, or does it represent Amazon's cut of those transactions?
--Dean
$200 is Amazon's cut. $2000 per day is the amount paid to Turkers.
The story I heard from an original mturk team member was that AMZN has no illusions of mturk being profitable any time soon. They think this kind of service will be big several years from now, and wanted to get in on the ground floor.
I spent $300 on Turk one day and that got me a personal customer service rep!
There seems to be two problems with this 1) undercounting HITs completed within your 24 hour observation cycle 2) overcounting the HITs which exist for over 24 hours in the system. How did you query the DB for the total value of the HITs, I've only seen total count on their site.
I visit the site every hour. I only count as arrival the first time that I see a given HITid.
I query my own database in which I collect data from Mechanical Turk, not the Mechanical Turk website.
If amazon's using it for tasks, the cost savings might easily be far greater than $2000 a day... making it worthwhile for them.
Michael, I agree that Amazon keeps MTurk around because it is useful for them and generates some buzz. (And I am glad that they keep it :-)
I'd guess you're right that the correct figure is at the same order of magnitude, but it might be one higher. Across about 20k HITs we've created the average is a little more than three assignments per HIT. For most research projects, which are the bulk of what we use MTurk for, two assignments is the norm. But there are tasks for which we've requested hundreds of assignments, and I do wonder how common those types of HITs are with other requesters.
Anyway, even at 10x your general point still stands.
@Michael:
My understanding is that Amazon created MTurk for its own internal purposes and later turned it loose on the public. They'd still be using it even if it were not available to other requestors.
Have you noticed Amazon has been hiring for Mechanical Turk lately?
www.job-search-engine.com/keyword/turk
Senior Manager
Technical Program Manager
Product Manager
Solutions Architect
After letting Mechanical Turk languish for the last couple years, it seems they're not going to be content with a few thousand dollars a month much longer.
AMT is so weird.
You also have to take into account the fact that only turkers from the US and India get cold hard cash out of mturk. Everybody else is getting gift certificate transfer and we have to spend our mturk money on amazon.com!
Good point, I have not thought of the extra benefit for Amazon when paying through gift certificates!
How come turkers in India can get cash?
I purchase some things from Amazon with the money I make. The rest goes to my bank account using Amazon Payments website. I imagine turkers in India just do a transfer also.
One issue I must point out, since there are a huge number of "hot" HITs you may never be able to see it on the website, it comes and goes in one second, I am pretty sure there are about 10 such kind of tasks, and my estimation is each of them has approximate 100 to 200 $ per day. A good example is "Content Review"'s HITs, you may try to use "previewandaccept" functionality, and you may happen to accept one but on dashboard you see nothing.
Actually based on our statistics and the power-law distribution of task durations, the amount of such "short" tasks is not that high. Perhaps 3-5% of the marketplace volume. We were initially crawling every 60 minutes, then every 15, and lately every 2 minutes.
The statistics for the marketplace are very consistent, although you would expect to see many more tasks when crawling every 2 minutes vs crawling every 60 minutes.
There is a certain bias when observing the market empirically as a human: Humans tend to observe specific tasks, ignoring the vast majority of "uninteresting" tasks.
I may not against the percentages you estimated about the short HITs, because their amount of awards are usually very small (1 cent or so), but I guess 2 minutes is still way too long, I had one experiment, crawling it every 4 seconds, looking at the very HIT I mentioned, they seldom last more than 4 seconds.
Just one interesting statistic from one of my friend used to work only on this HIT with a script to automatically accept it. He make $ 40 daily. Given that he works only 10 hours, and there may be more than one assignment per HIT, I estimate this HIT will throw more than $100 onto the market.
I just want to point out, there is way to accept HIT automatically even before it appears on the list, therefore they never show up.
Post a Comment