Page 128 -

114 CHAPTER 5 Surveys

a website or social networking), or snowball recruiting (where respondents recruit
other potential respondents) (Müller et al., 2014).
It is important to note that different academic communities have different stan-
dards in how they apply sampling techniques. For instance, there are many people
in the fields of social science and statistics who believe that without strict random
sampling, no survey data is valid (Couper, 2000; Sue and Ritter, 2007). On the other
hand, the HCI community has a long history of using surveys, in many different
ways, without random sampling, and this is considered valid and acceptable. Part
of this difference may stem from the nature of research in different communities.
In some research communities, large national and international data sets are col-
lected using rigorous, structured sampling methodologies. The general social survey
in the United States (gss.norc.org) and the National Centre for Social Research in the
United Kingdom (http://www.natcen.ac.uk/) are examples in the fields of sociology
and public policy. Researchers can take these high-quality, probability-sampled data
sets and perform analyses on the many variables in them. This is not the model of re-
search used in HCI. In HCI, researchers must, typically, collect the data themselves.
No large, well-structured data sets exist. The HCI researcher must go out, find users
to take part in their research, and collect the data, as well as analyze the data. Because
of this difference, both probability samples and nonprobability samples are consid-
ered valid in HCI research. There are a number of techniques for ensuring validity in
nonprobability-based samples. The next sections detail the standard approaches for
ensuring validity in nonprobability-based samples.
It is also important to note that, very often, surveys are used by HCI researchers,
in conjunction with other research methods, when there is no claim of the representa-
tiveness of the survey responses, in fact, it is openly acknowledged that the responses
represent a convenience sample. This is quite common, so, for instance, if you look
at recent papers from the CHI conference, not only will you find surveys with over
1000 responses (such as Moser et al., 2016; Chilana et al., 2016), you will also find
papers that combine small surveys with other research methods such as diary studies
(Epstein et al., 2016), interviews (Dell and Kumar, 2016), usability testing (Kosmalla
et al., 2016), and log analysis (Guy et al., 2016). These examples only scratch the
surface; clearly, small, nonprobabilistic samples are used throughout HCI research
on a regular basis, without concern.

5.5.1 DEMOGRAPHIC DATA

One way of determining the validity of survey responses is to ask respondents for a
fair amount of demographic data. The goal should be to use the demographic data to
ensure that either the responses represent a diverse, cross-section of respondents or
the responses are somewhat representative of already-established, baseline data (if any
exists). For instance, even basic demographic data on age, gender, education, job re-
sponsibility, or computer usage can help establish the validity and representativeness of
survey responses when respondents are self-selected (Lazar and Preece, 2001). While
this is not equivalent to the validity of a population estimate or random sampling, it is
better than no check on the validity or representativeness of survey responses. Note