Using Mechanical Turk for research
After Kittur, Chi, and Suh’s Crowdsourcing user studies with Mechanical Turk (CHI 2008), Ed Chi and I decided to run another study using Amazon’s Mechanical Turk service. I finally tallied our “expenses” and response rate, and though it would be interesting to share.
Now, Ed and I were interested in learning something interesting about web search, but given the *mass* of work that has been done in this area before, we set out cautiously. I am a proponent of looking carefully at people’s real behaviors (ethnography-style), while Ed wanted have a larger sample than is usually possible with ethnographies. Therefore, I constructed a pretty detailed critical-incident style questionnaire (which was consequently fairly long), asking people to provide details about (ONLY) their most recent search experience. We probed for people’s actions, social interactions, and context before, during, and after the actual search act, hoping to learn about how people made use of social inputs/exchanges to help in the process of their search.
There were 27 total questions: some were check boxes or Yes/No, but most were open text boxes. We started out paying $0.20 per complete questionnaire and increased that to $0.35 towards the end when we wanted to gather responses more quickly. Still, we put this survey out on Mechanical Turk from March 7–April 9 (almost exactly 1 month). Here’s how we fared:
Total submitted responses: 164
Total responses that I accepted and compensated: 157 (95.7% of the submitted responses)
Total accepted responses that were actually usable (good): 150 (95.5% of the accepted responses).
Total paid: $41.50
Total time spent analyzing the data: wouldn’t it be cool if I knew that?!
This comes to paying $0.28 on average for one those 150 good responses. Now, those responses were not just good, they were great! We analyzed the search experiences described by those 150 people to construct a (preliminary) model of social search.
…And ever since, I’ve been stuck on this question of what the heck we even mean by “social”? Our papers take a broad stance on what social search is, but I am very interested in gathering examples of a variety of social search experiences, and conversely, experiences that are explicitly NOT SOCIAL (or not social search).
————–
Our workshop paper: Towards a Model of Understanding Social Search (presented at the JCDL Workshop on Collaborative Information Retrieval in Pittsburgh, PA, June 2008)
Longer version of the paper will be presented at CSCW in November 2008 in San Diego, CA.
No Comments Yet