Page 120 -

106 CHAPTER 5 Surveys

full evaluation involving user-based testing (described more in Chapter 10) (Müller
et al., 2014). Since surveys primarily rely on users to self-administer, remember data
that occurred in a previous point in time, and return the survey, without a researcher
being physically present, there are a lot of background details that must receive at-
tention for the data collected to be valid and useful.
Is a survey the same thing as a questionnaire? Well, many people do use the two
terms interchangeably. Others differentiate between the “questionnaire,” which is the
list of questions, and the “survey,” which is the complete methodological approach,
including sampling, reminders, and incentives. For instance, Dillman states clearly
that “the questionnaire is only one element of a well-done survey” (Dillman, 2000,
p. 149). While we acknowledge the difference, since the two terms are often used
interchangeably, we use them interchangeably in this chapter.

5.2 BENEFITS AND DRAWBACKS OF SURVEYS

Surveys have many benefits and a few drawbacks. Using a survey, it is easy to col-
lect data from a large number of people, at a relatively low cost. Surveys can be
used for many different research goals. Because they allow access to a large number
of people, surveys can be very useful for getting an overview, or a “snapshot,” of a
user population. Surveys do not require advanced tools for development; they can be
distributed easily using e-mail or existing survey websites, or done on paper. From
a practical point of view, surveys are among the research methods most likely to get
approval from an institutional review board or human subjects board because they
are relatively unobtrusive (see Chapter 15 for more information on institutional re-
view boards).
There are a few drawbacks to using surveys as a research method. A survey is very
good at getting limited “shallow” data from a large number of people but is not very
good at getting “deep,” detailed data. Since surveys are typically self-administered
(either on paper, e-mail, or websites), if interesting phenomena start appearing, it is
usually not possible to ask follow-up questions, or go back and change the original
survey instrument to ask more detailed questions.
Another major drawback is that surveys can sometimes lead to biased data when
the questions are related to patterns of usage, rather than clear factual phenomena.
For instance, a question such as the user's age or gender is not subject to interpre-
tation or memory. Clearly, on a given day, an individual has an age (say, 33 years
old) and a gender (male). However, questions related to mood (e.g., “How were you
feeling when you were using this software application?”) are subject to recall bias
if the event took place a significant amount of time earlier. Another example might
be to ask people to recall how much money they have spent on e-commerce within
a 6-month period or how many times they completed a certain task using a specific
software application. Their response might be biased and either overestimate or un-
derestimate the amount (Andrews et al., 2003). If data is of a factual nature and can
instead be collected in an automated fashion using a computer, it may be a preferred