Page 77 -

62 CHAPTER 3 Experimental design

learning effect and fatigue push the observed value in opposite directions and the
combined effect is determined by the specific context of the experiment. If the tasks
are simple and less susceptible to the learning effect, but tedious and long, the impact
of fatigue and frustration may outweigh the impact of the learning effect, causing
participants to consistently underperform in later sessions. If the tasks are compli-
cated and highly susceptible to the learning effect, but short and interesting, the im-
pact of the learning effect may outweigh the impact of fatigue, causing participants
to consistently perform better in later sessions.
The instructions that participants receive play a crucial role in an experiment
and the wording of the experiment instructions should be carefully scrutinized be-
fore a study. Slightly different wording in instructions may lead to different par-
ticipant responses. In a reported HCI study (Wallace et al., 1993), participants were
instructed to complete the task “as quickly as possible” under one condition. Under
the other condition, participants were instructed to “take your time, there is no rush.”
Interestingly, participants working under the no-time-stress condition completed the
tasks faster than those under the time-stress condition. This suggests the importance
of critical wording in instructions. It also implies that the instructions that partici-
pants receive need to be highly consistent. When a study is conducted under the
supervision of multiple investigators, it is more likely that the investigators give in-
consistent instructions to the participants. Instructions and procedures on a written
document or prerecorded instructions are highly recommended to ensure consistency
across experimental sessions.
Many times, trivial and unforeseen details introduce biases into the results. For
instance, in an experiment that studies data entry on a PDA, the way the PDA is physi-
cally positioned may have an impact on the results. If no specification is given, some
participants may hold the PDA in one hand and enter data using the other hand, other
participants may put the PDA on a table and enter data using both hands. There are no-
table differences between the two conditions regarding the distance between the PDA
screen and the participant's eyes, the angle of the PDA screen, and the number of hands
involved for data entry. Any of those factors may introduce biases into the observed
results. In order to reduce the biases attributed to experimental procedures, we need to
• randomize the order of conditions, tasks, and task scenarios in experiments that
adopt a within-group design or a split-plot design;
• prepare a written document with detailed instructions for participants;
• prepare a written document with detailed procedures for experimenters; and
• run multiple pilot studies before actual data collection to identify potential biases.
A pilot study is not a luxury that we conduct only when we have plenty of time
or money to spend. On the contrary, years of experience tells us that pilot studies are
critical for all HCI experiments to identify potential biases. No matter how well you
think you have planned the study, there are always things that you overlook. A pilot
study is the only chance you have to fix your mistakes before you run the main study.
Pilot studies should be treated very seriously and conducted in exactly the same way
as planned for the actual experiment. Participants of the pilot study should be from