Page 286 -

10.5 The process of user-based testing 275

recruitment in usability testing is generally seen to be more flexible than in experi-
mental design, and samples of convenience are common and appropriate (Tullis and
Albert, 2008). While it is very important that the recruited participants accurately
represent the target user population, it is less relevant how you recruit those users.
Unless you are dealing with multiple user populations across cultures, countries, or
languages (in which case you may want to do usability testing at each site), it can be
satisfactory, for instance, to recruit users from only one or two companies or in only
one geographic area.

10.5.3 HOW MANY USERS ARE SUFFICIENT?
One of the most common questions when planning usability testing is “how many us-
ers do I need to have?” It’s also a bit of a hotbed of discussion in the HCI community,
and a consensus has not emerged over time. If you were doing a strict experimental
design, the types of research design and the statistical tests that you run would dictate
the minimum number of participants required. However, usability testing has differ-
ent goals and different requirements.
Many people say that five users is sufficient, and that five users will find ap-
proximately 80% of usability problems in an interface (Virzi, 1992). This has be-
come an often-quoted number in HCI, but many other researchers disagree with
the assertion. The major challenge in determining the right number of users, is that
you don’t know in advance how many interface flaws exist, so any estimate of how
many users are needed to find a certain percentage of interface flaws is based on the
assumption that you know how many flaws exist, which you probably don’t. Other
research studies have found that five users are not sufficient to discover and iden-
tify a majority of usability flaws (Lindgaard and Chattratichart, 2007; Spool and
Schroeder, 2001). In a classic paper, Nielsen and Landauer, who in earlier work had
asserted the number five, expressed that the appropriate number depends on the size
of the project, with seven users being optimal in a small project and 15 users being
optimal in a medium-to-large project (Nielsen and Landauer, 1993). However, in
that same paper, they indicated that the highest ratio of benefits to costs is when you
have 3.2 users doing usability testing (Nielsen and Landauer, 1993). In an analysis
of existing research on the topic, Hwang and Salvendy (2010) suggest that 10 ± 2
is the optimal number of users for usability testing, although more recent work by
Schmettow (2012) suggests that even 10 users is not enough to discover 80% of the
usability problems.
Lewis says that all authors could theoretically be right about the appropriate num-
ber of users, as it depends on how accurate they need to be, what their problem
discovery goals are, and how many participants are available (Lewis, 2006). Even
if five users are enough, what happens when you have multiple user groups taking
part in usability testing. Do you need five users from each group? Lindgaard and
Chattratichart (2007) take a different approach: they assert that the number of us-
ability flaws found depends more on the design and scope of the tasks, rather than
on the number of users.