Page 302 -

10.5 The process of user-based testing 291

USABILITY TESTING AT FIDELITY INVESTMENTS
Usability testing took place at Fidelity Investments, evaluating the prototype
of an interface to manage individual benefits, including company benefits,
retirement savings, and pensions. A total of 27 participants tested the first
prototype (this included both younger and older users, which would be expected
for this type of interface). Each participant was given 15 tasks to complete, such
as switching retirement money from one plan to another and determining what
pension benefits would amount to if the individual retired at 65 years old. Task
success rates were 64.2% for users under 55 years old and 44.8% for users aged
55 years or older.
Based on the usability testing of the first prototype, changes were made
to the interface, including improved terminology, making links consistently
obvious, adding more instruction for detailed table data, adding more titles, and
removing false window bottoms and mouseover-based navigation tabs.
Usability testing then took place with the second interface prototype and a
new set of 22 participants took part. The new participants had the same profile
of age and computer experience as the participants in the first round of testing.
The participants in the second round of usability testing were given the same
15 tasks as participants in the first round of testing, with a few minor wording
changes due to the updated interface. With the new interface prototype, task
success rates improved to 80.6% for users under age 55 and 58.2% for users
aged 55 years and older (Chadwick-Dias et al., 2003).

Unlike in other traditional research, in usability testing, it is considered a standard
practice to tell participants before they start that they are not being tested. Rather,
the participants are testing the interface. Their feedback is important. They are the
experts and have the right to criticize the interface. Users are not being tested. You
may need to remind them of that fact multiple times.
Note that during the testing session, there are two “tracks” of data collection.
One track is the quantitative metrics, such as task and time performance. Moderators
may be timing the tasks, data logging may be keeping track, or video recording
may be used for later review. The second track is the qualitative data. Sometimes,
participants are very talkative and provide a verbal track of what they are doing. If
the participants are not very talkative, and if thinking aloud is the methodological
goal, moderators should try to encourage them to share more of how they are feeling.
However, these reminders should not be often, since the more that the moderator in-
terrupts the user, the more the user feels watched, the more the user’s cognitive flow
is interrupted, and the more that the user’s behavior may deviate from normal. The
thinking aloud protocol is more common in formative usability testing than in sum-
mative usability testing, since, if quantitative metrics are considered very important
by the stakeholders of that interface, the more the participant stops to talk, the more
that their task time is interrupted. So while it is acceptable for the users to stop every