Page 284 -

10.5 The process of user-based testing 273

interfaces (West and Lehman, 2006). Sauro and Lewis (2012) have a similar view,
describing any type of usability test to find and fix usability problems as formative,
and describe summative only as metrics for describing usability.
The one thing that most authors agree on is that earlier, formative usability tests
tend to focus more on qualitative feedback, moderator observation, and problem dis-
covery, whereas summative usability tests tend to focus more on task-level mea-
surements, metrics, and quantitative measurements (Lewis, 2006). The “Usability
Testing of the Kodak Website” sidebar gives an example of formative and summative
usability testing.

USABILITY TESTING OF THE KODAK WEBSITE

The Eastman Kodak Company is one of the world’s largest manufacturers and
marketers of imaging products. Both formative and summative usability testing
took place on the Kodak website.
Formative testing took place on a paper prototype of the new home page
design, specifically the links and groups. Twenty participants were given 30
tasks and were asked to identify the homepage link most likely to lead to the
information that would complete that task. Participants were then asked to
describe what type of content they expected to find behind each homepage
link. Finally, participants were given descriptions of what actually was behind
each home page link, and were asked to rate how well the label matched the
actual content.
Later, summative testing with 33 participants took place on a working
prototype of the new home page and all top-level pages on the site. A list of
22 tasks was developed, but each participant was given only 10 information-
seeking tasks to complete. Some tasks were attempted by all 33 participants,
while other tasks were attempted by only 11 participants. All links were
functional, although not all visual design elements on the pages were complete.
Each participant was given a maximum of 3 minutes to complete each task.
Task completion for individual tasks ranged from 100% to 9% in the allotted 3
minutes. Based on the results of the usability testing, changes were made to the
pages, including removing images along the left side of the page, adding longer
descriptors to more of the links, and labeling major chunks of information
(Lazar, 2006).

Whether a usability test is formative, summative, or validation can influence
how formal or informal the usability test is. At one end of the spectrum is a for-
mal approach to usability testing, which parallels experimental design. This form of
usability testing can involve specific research questions, research design (between-
subject design or within-subject design), and multiple interfaces to test. If you are
using inferential statistics, hypotheses, a control group, large numbers of subjects,