Page 439 -

430 CHAPTER 14 Online and ubiquitous HCI research

To understand the challenges, we might compare crowdsourced studies to tradi-
tional studies. Familiar lab-based studies use advertisements and word of mouth to
spread the word, often offering a small honorarium to encourage interest. Participants
come to the lab, spend some amount of time—perhaps an hour or two—and are given
payment upon completion of their participation. Although this approach often leads
to maddening difficulties in recruiting sufficient numbers of participants, it offers
several advantages. Perhaps most importantly, individuals who express interest in
such studies can usually be depended upon to complete the studies appropriately and
in good faith. Enticements of $20 or even $50 might be sufficient to encourage some
people to participate in studies that do not interest them, but it is generally not worth
the bother to participate without taking the study seriously. Although we have not
evaluated the question empirically, our experience has been that most people who
agree to join in lab studies do so honestly and with every intention of working with
the researcher to meet the goals of the study.
Direct interaction with participants is a second, closely related, benefit. When
someone sits down in your lab to participate in a study, you will be able to talk with
them and to observe their work as they complete the tasks at hand. These interactions
provide valuable “sanity check” information, allowing you to form impressions of
each individual's task performance and motivations, and specifically to avoid partici-
pants who might not be taking your tasks seriously. You certainly do not want to rush
to discard data from someone who is goofing off—including the data and raising the
concern in a discussion would be much more appropriate—but having observed this
behavior might help you understand results, particularly if you identify participants
with bad behavior that might have led to unexpected results in your data.
There are many appealing aspects to the use of human computation in HCI re-
search. A properly constructed human computation study can be constructed in soft-
ware, deployed on a web site (often using dedicated software services, as discussed
later), and advertised to large numbers of potential workers at reasonably low cost.
Participant enrollment, completion of consent forms, administration of the study, and
data collection can be largely automated, thus eliminating the need for tedious work
that has afflicted many graduate and undergraduate student workers. Online human
computation studies can also enroll many more participants than comparable tradi-
tional studies, providing greater statistical power. The user base may be large and
diverse, involving a broader range of education levels, ethnicities, and backgrounds
than you would likely get in a lab (Kittur and Kraut, 2008).
Of course, the reality is somewhat more complicated. As with any other type
of HCI study, human computation experiments require careful selection of partici-
pants and tasks. You will also need an appropriate software infrastructure, capable
of handling all of the enrollment and screening processes conducted to enroll par-
ticipants; and the presentation of tasks and collection of data necessary for the study
itself. Human computation studies must be carefully designed to ensure high-quality
responses: although tasks involving intrinsic motivation such as entertainment, in-
tellectual curiosity, or accessing a desired resource might motivate participants to