Page 80 -

3.6 Experimental procedures 65

Both physical and social environmental factors may introduce systematic er-
rors into the observed data. For instance, a study that examines the performance of
a speech-recognition application may yield lower recognition error rates than the
actual value if there is a significant level of ambient noise during the experiment
session. Even when the study investigates applications other than speech, loud en-
vironmental noise may distract the participants or induce fatigue. Regarding social
factors, a participant with a person watching over his shoulder may perform differ-
ently from a participant who is seated alone. Environmental factors may cause more
problems when the experiment is not conducted in a lab, but in locations such as the
participant's home or workplace. The following guidelines can help us avoid or con-
trol environment-induced biases:
• In a lab setting, make sure the room is quiet, the lighting is appropriate, and the
chairs and tables are comfortable. The room should be clean and tidy, without
notable distractions.
• Whenever possible, the participant should be seated alone and the experimenter
can observe the session from another room via a one-way mirror or monitors.
• In a field study, the experimenters should visit the location before the scheduled
time to confirm that the setting meets the requirements of the study.
Finally, it is important to realize that, no matter how hard you try to avoid biases,
they can never be completely eliminated. A well-designed experiment with lots of
consideration for controlling bias can improve the data, making the observed results
closer to the actual values, but still subject to the impact of biases. Therefore, we
should be careful when reporting the findings, even when the study results are sta-
tistically significant.

3.6 EXPERIMENTAL PROCEDURES

Experiments are conducted in dramatically different fields to answer a myriad of
questions. Experiments in the HCI field, similar to many studies in sociology or
psychology, typically involve human subjects. Studying human subjects is quite dif-
ferent from studying metal or plant reactions, or other animals, and introduces many
interesting issues or challenges. The concerns and practices of working with human
subjects are discussed in detail in Chapter 15. In this section, we briefly introduce the
procedures for experiments that study human subjects.
In the lifecycle of an HCI experiment, we typically go through the following process:

1. Identify a research hypothesis.
2. Specify the design of the study.
3. Run a pilot study to test the design, the system, and the study instruments.
4. Recruit participants.
5. Run the actual data collection sessions.
6. Analyze the data.
7. Report the results.