Mechanical Turk, Human Subjects, and IRB's

Academics that engage in user studies have often to apply to an IRB (Institutional Review Board) to get permission to do research with human subjects. These applications are mainly targeting researchers in biomedical sciences, but to be on the safe side many universities also require IRBs for usability studies, or any other research that obtains personally identifiable information from the participants.

So, the question is: Does someone need to apply for an IRB to use Mechanical Turk? The answer is "it depends". My own take is that for the majority of the tasks posted on MTurk, an IRB is not required. An IRB is required only when someone studies the Turkers, not when postings tasks to be completed by Turkers.

Here is what I have found from 1 and 2

Am I proposing Human Subjects Research?

Research is considered to involve human subjects when an investigator conducting research obtains (1) data through intervention or interaction with a living individual, or (2) identifiable private information about a living individual.

(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains:
(1) Data through intervention or interaction with the individual, or
(2) Identifiable private information.

In the case of Mechanical Turk, we do not retrieve any identifiable private information about a living individual. So, to engage in human subjects research, we need to collect data data through intervention or interaction with a living individual.

Let's see the corresponding definitions for intervention and interaction.

Intervention, as it pertains to research involving human subjects

defined in 46.102 within the Human Subject definition – includes both physical procedures by which data are gathered (for example, venipuncture) and manipulations of the subject or the subject's environment that are performed for research purposes

Interaction, as it pertains to research involving human subjects

defined in 46.102 within the Human Subject definition – includes communication or interpersonal contact between investigator and subject

Intervention: My own take is that we do not have any intervention. We do not physically interact with the Turkers, and we do not modify the environment of the subjects for research purposes.

Interaction: My own take is that we do not have any interaction with the Turkers either as we simply post the tasks and ask them to complete them. If this is considered interaction, we could classify as interaction any visit to our own web pages.

Of course, the above are my own points of view. I am not a lawyer, so my interpretations above may be wrong. Judge yourself if my interpretations are correct or not. (And let me know in the comments if you disagree.)

I also found some additional information is available from San Francisco State University.

Content Experts/Consultants/Key Informants

It may not necessary to get human subjects approval if interview questions are with experts about a particular policy, agency, program, technology, technique, or best practice. The questions are not about the interviewee themselves, but rather about the external topic. For instance, questions will not include demographic queries about age, education, income or other personal information.

Human Subjects review will be required when a researcher is interviewing individuals about content, but there is a research question or hypothesis involved, or an “agenda.” The researcher intends to analyze and generalize the results, that is, look for common themes in the collected data, try to universalize the interviewees’ experiences, or quantify the results in some way.

Examples of content expert projects that may not require human subjects review:

In all the following examples, the questions are focused on the facts about the program, policy, software, curriculum, procedures or project. The researcher will simply report the facts as they are related by the content experts. You may not need to submit a protocol or an informed consent form for human subjects approval if:

• you are interviewing managers in a company about their billing procedures, or their use of a particular software program, or
• you are interviewing or surveying teachers about what should be included in the development of a particular curriculum unit, or
• you are interviewing entrepreneurs about the obstacles they faced in starting their own businesses, and how they overcame them, or
• you are asking a panel of nurses and doctors to review your antismoking program for teens for correct medical content, or
• you are interviewing social agency directors about their client intake procedures.

So, if you are doing some annotation work, where you ask not for personal opinions, but instead you ask Turkers (or any user actually) to tell you something about the "true state of the world", then an IRB is not required. If however you ask for personalized experiences (and you happen to be in academia) then it seems that an IRB may be required.