Page 78 -

3.5 Reliability of experimental results 63

the target population. Having one or two members from the research team complet-
ing the designed tasks is not a pilot study in its true sense (Preece et al., 1994).

3.5.2.3 Bias caused by participants
Many characteristics of the participants may introduce systematic errors into the re-
sults. Potential contributors may be in a specific age range or have particular com-
puter or internet experience, domain knowledge, education, professional experience
and training, or personal interests. For instance, if we are running an experiment to
test the user interface of a new mobile phone model, we might recruit participants by
posting announcements on a popular blog on https://www.cnet.com. Since this web-
site features highly technical news and reviews related to information technology, its
visitors normally have a strong technical background and rich experience in using
IT devices. As a consequence, the observed data would tend to outperform what we
would observe from the general public. The following guidelines can help us reduce
systematic errors from the participants:
• Recruit participants carefully, making sure the participant pool is representative
of the target user population (Broome, 1984; Smart, 1966).
• Create an environment or task procedure that causes the least stress to the users.
• Reassure the participants that you are testing the interface, not them, so they are
calm and relaxed during the experiment.
• Reschedule a session or give participants some time to recover if they arrive
tired, exhausted, or very nervous.

3.5.2.4 Bias due to experimenter behavior
Experimenter behavior is one of the major sources of bias. Experimenters may in-
tentionally or unintentionally influence the experiment results. Any intentional ac-
tion to influence participants' performance or preference is unethical in research and
should be strictly avoided. However, experimenters may unknowingly influence the
observed data. Spoken language, body language, and facial expressions frequently
serve as triggers for bias. Let us examine the following scenarios:
1. An experimenter is introducing an interface to a participant. The experimenter
says, “Now you get to the pull-down menus. I think you will really like them.…
I designed them myself!”
2. An experimenter is loading an application for a participant. The response time is a
bit long. The experimenter is frustrated and says, “Damn! It's slower than a snail.”
3. An experimenter is loading an application for a participant. The response time
is a bit long. The experimenter waits uneasily, tapping fingers on the desk and
frequently changing body position while staring at the screen impatiently.
4. A participant arrives on time for a study scheduled at 9 a.m. The experimenter
does not arrive until 9:10 a.m. After guiding the participant into the lab, the
experimenter takes 10 minutes to set up all the equipment. Once the experiment
starts, the experimenter finds that the task list is missing and runs out of the lab
to print a copy.