Page 305 -

294 CHAPTER 10 Usability testing

types of technology could be successfully used in these settings by these users. The
technologies themselves are not the interfaces being tested for usability. Technology
probes have been used to understand how family members communicate and share
images (Hutchinson et al., 2003) and how people in a relationship show public af-
fection (O’Brian and Mueller, 2006). The focus in a technology probe isn’t the probe
itself but, rather, what can be learned about the people taking part and what technolo-
gies they could potentially use.
11
A Wizard-of-Oz method is essentially a simulation of functionality that doesn’t
exist yet in an interface application. The user perceives that they are interacting with
the actual interface and system. In reality, the user is interacting with another hu-
man being that is providing the responses to the user (Dahlback et al., 1993; Gould
et al., 1983). Wizard-of-Oz methods can be used when the functionality has not been
built due to cost concerns and when the technology doesn’t exist, to test potential
future interfaces (White and Lutters, 2003). In addition, due to the low time and
cost involved, the method may also be helpful in determining feasibility and testing
concepts prior to any real systems development (White and Lutters, 2003). Because
there can sometimes be a time delay before the “wizard” responds, it can be helpful
to have a set of precompiled responses that can quickly be accessed, which helps to
improve the realism of the simulation (since participants typically don’t know that
the functionality isn’t being provided by the computer system). The Wizard-of-Oz
method has been used in evaluating motion-based computer games for children
(Höysniemi et al., 2004), spoken dialog systems in driving vehicle simulators (Hu
et al., 2007), and speech recognition systems (Sinha et al., 2001).

10.7 SUMMARY
Usability testing is often known more generally as “user research.” Usability testing,
typically involves representative users attempting representative tasks in representa-
tive environments, on early prototypes or working versions of computer interfaces,
with the goal of improving the quality of an interface by finding flaws, areas of the
interface that need improvement. In reality, the approaches utilized in usability test-
ing are often the same as those used in classic research. Metrics utilized in usability
testing include measurement of task performance and time performance, similar to
experimental design, but usability testing often has different end goals. The goal is
not to create research that can be generalized to other projects, but rather, to dis-
cover specific flaws so that a specific interface can be improved. As an example,
making immediate changes to the interface allows for those changes to be evaluated
during the next user test, which is considered acceptable in usability testing, but
would be considered unacceptable in experimental design. While expert reviews and
automated usability testing do help improve interfaces, typically they are not con-
sidered HCI research and/or user research, since they do not involve representative

1
The name comes from the man behind the curtain in the movie The Wizard of Oz.