Page 79 -

64 CHAPTER 3 Experimental design

In Scenario 1, the experimenter is very demanding and the comment may make
the participant reluctant to provide negative feedback about the interface in case it
hurts the experimenter's feelings. Therefore, the data collected from the participant,
especially the subjective data, are likely to be better than the actual value. In Scenarios
2 and 3, the experimenter's spoken language or body language reveals negative at-
titude toward the application. Participants would register those cues and would form
a negative perspective even before their first encounter with the application and the
collected subjective ratings and feedback would be biased against the application. In
Scenario 4, the unprofessional and slack style of the experimenter would give a nega-
tive impression to the participant, which may impact the participant's performance as
well as the subjective ratings and feedback.
When multiple experimenters are involved in the experiment, bias is likely to
occur due to inconsistency in instructions and training, as well as individual styles
and attitudes. If one of the experimenters is very patient, offers long training ses-
sions, and demonstrates all related commands to the participants before the actual
task, while the other experimenter is pushy, offers shorter training sessions, and only
demonstrates a subset of the commands, the participants who complete the experi-
ment under the guidance of the first experimenter may systematically outperform the
participants who complete the experiment with the second experimenter. In order to
control possible biases triggered by experimenters, we need to
• Offer training opportunities to experimenters and teach them to be neutral, calm,
and patient when supervising experiments.
• Make sure that the experimenter arrives at least 10 minutes before the scheduled
sessions and gets everything ready before the session starts.
• Whenever possible, have two experimenters supervise a session together, one
as the lead experimenter and the other as the assistant experimenter. The lead
experimenter is responsible for interacting with the participants. The assistant
experimenter observes the session closely, fixes errors if noted, and takes notes
when necessary.
• Prepare written documents with detailed procedures for experimenters and
require all experimenters to follow the same procedure strictly.
• When appropriate, record important instructions before the experiment and
play the recording to the participants during the experiment. In this way, we can
guarantee that all participants go through the same training process and receive
the same instructions.
3.5.2.5 Bias due to environmental factors
Environmental factors play an increasingly important role in HCI research due to
the rapid development in mobile computing, universal accessibility, and recognition-
based technologies. Environmental factors can be categorized into two groups: phys-
ical environmental factors and social environmental factors. Examples of physical
environmental factors include noise, temperature, lighting, vibration, and humidity.
Examples of social environmental factors include the number of people in the sur-
rounding environment and the relationship between those people and the participant.