Page 466 -

458 CHAPTER 15 Working with human subjects

Additional care is necessary when study designs require multiple groups that dif-
fer in some dimension. Ideally, the groups would differ in the relevant attribute but
be comparable in all others. Any other differences would be possible confounding
variables—factors that could be responsible for observed differences. In the study of
gender differences in information management, the male and female groups should
be comparable in terms of education, age, income, professional experience, and as
many other factors as possible. If the women were significantly younger than the
men, it might be hard to determine whether any performance differences were due to
age or gender: further experimentation may be necessary.
Although these issues may be most important for controlled experiments, the
identification of an appropriately general group of participants is always a challenge.
Appropriate recruiting methods can help, but there are no guarantees. Despite your
best efforts to find a representative population, you always face the possibility that
your group of participants is insufficiently representative in a way that was unan-
ticipated. As this bias is always possible, it is best to explicitly state what steps you
have taken to account for potentially confounding variables and to be cautious when
making claims about your results.

15.2.2 HOW MANY SUBJECTS?
Determining the number of participants to involve in a research study is a trade-off
between the information gained in the study and the cost of conducting it. Studies
with a very large number of participants—say, tens of thousands—probably involve
many people of different ages, educational backgrounds, and computer experience.
Any outcome that you see consistently from this population may therefore not be
something that can be explained away by the specific characteristics of the individual
participants: it is likely to be a “real” effect. Huge studies like this are particularly
helpful for controlled experiments in search of statistically significant results. Even
subtle differences can be statistically significant if the populations are sufficiently
large.
Unfortunately, large studies are difficult and expensive to run, involving substan-
tial costs for recruiting, enrolling, conducting the study, and managing data. If the
participants are not at your workplace, there may be travel involved, and many stud-
ies pay people for their time. If your study allows you to involve many people at
once—perhaps 20 people in a roomful of computers—you may be able to achieve
some efficiencies in terms of the time involved. However, research that involves one-
on-one interactions between a researcher and a participant may have costs that grow
linearly with the number of participants.
At the other extreme, a study with one individual has very real limitations. This
study would be relatively inexpensive, but also very limited. Because this study
would not have a range of users with different characteristics, any results would run
the risk of telling you more about the participant than they did about the research
question at hand. If you're conducting an ethnographic study (Chapter 9) with one
person, you may learn a great deal about how that person performs certain types of