Page 81 -

66 CHAPTER 3 Experimental design

Within a specific experiment session, we typically go through the following steps:
1. Ensure that the systems or devices being evaluated are functioning properly,
the related instruments are ready for the experiment.
2. Greet the participants.
3. Introduce the purpose of the study and the procedures.
4. Get the consent of the participants.
5. Assign the participants to a specific experimental condition according to the
predefined randomization method.
6. Participants complete training tasks.
7. Participants complete actual tasks.
8. Participants answer questionnaires (if any).
9. Debriefing session.
10. Payment (if any).

Some experiments may require more complicated steps or procedures. For exam-
ple, longitudinal studies involve multiple trials. We need to make sure that the tasks
used in each trial are randomized in order to control the impact of the learning effect.
A number of open source platforms have been developed to help researchers design
experiments, collect data, and analyze the results. One example is the Touchstone experi-
mental design platform. The Touchtone system includes a “design” platform for examin-
ing alternative, controlled experimental designs, a “run” platform for running subjects, and
an “analysis” platform that provides advices on statistical analysis (Mackay et al., 2007).

3.7 SUMMARY
Experiment design starts with a clearly defined, testable research hypothesis. During
the design process, we need to answer the following questions:
• How many dependent variables are investigated in the experiment and how are
they measured?
• How many independent variables are investigated in the experiment and how are
they controlled?
• How many conditions are involved in the experiment?
• Which of the three designs will be adopted: between-group, within-group, or
split-plot?
• What potential bias may occur and how can we avoid or control those biases?
When an experiment studies only one independent variable, we need to choose
between the between-group design and the within-group design. When there is more
than one independent variable, we need to select among the between-group design,
the within-group design, and the split-plot design.
The between-group design is cleaner, avoids the learning effect, and is less
likely to be affected by fatigue and frustration. But this design is weaker due