Page 467 -

15.2 Identifying potential participants 459

work, but you have no idea about how representative the person’s habits are: you may
get unlucky and find someone who is completely unlike colleagues in the field. As
studies with few participants rarely, if ever, produce statistically significant results,
the conclusions that you can draw from these small studies are extremely limited.
Controlled experiments or empirical studies require a sample group of participants
large enough to produce statistically significant results. The research design (the number
of independent variables, within or between subjects) will play a role as well. Experiments
involving larger numbers of independent variables and between-subjects (as opposed to
within-subjects) experiments can require more participants (see Chapter 3). Limitations
on resources can often lead researchers to substitute the feasible experiment—the design
that requires fewer participants—for the experiment they'd prefer to be doing. In some
cases, statistical techniques can be used to determine the minimum number of subjects
necessary for a result of a given significance (Chapter 3). Usually, you want at least
15–20 participants: smaller studies may miss potentially interesting results.
The inclusion of more participants gives you more statistical power. As each par-
ticipant comes with costs in terms of time, energy, and money, there are always good
arguments in favor of limiting the size of the study. However, larger populations—
ranging from several dozen to several hundred participants—offer the possibility of
stronger statistical significance or the identification of subtle effects that would not
be significant in smaller populations.
Statisticians have developed a range of techniques for determining the number of
participants necessary for establishing statistically significant effects with differing
degrees of confidence: Cook and Campbell (1979) is a classic text in this area. These
techniques can help you understand how many participants you need before your
study starts, thus minimizing the chances for painful problems further down the line.
By contrast, case studies and ethnographic studies (Chapters 7 and 9) can often
be conducted with a small number of users. If your goal is to gather requirements
from domain experts, in-depth discussions with two or three motivated individuals
may provide a wealth of data. The length of the session also plays a role here: ethno-
graphic observations generally take more time per participant—and therefore place
more demands upon the participants—than controlled experiments.
Usability studies can also be successfully conducted with a small set of partici-
pants. These studies may use a combination of expert reviewers equipped with guide-
lines and heuristics, followed by user-based testing, to identify potential usability
problems with proposed interface designs (Chapter 10). Although early work in this
area was interpreted to mean that studies involving as few as five participants might
be sufficient for finding 2/3 of usability problems (Nielsen and Molich, 1990), this
claim has been the subject of significant debate, with more recent work suggest-
ing that significantly more participants might be necessary for effective coverage
(Hwang and Salvendy, 2010; Schmettow, 2012). User skills and background can play
an important role in determining the number of evaluators needed: as evaluators with
experience both in usability and in the problem domain can be more effective, fewer
numbers of so-called “double experts” may be needed (Nielsen, 1992). Of course,
these highly skilled participants can be incredibly hard to find and enroll.