Page 67 -

52 CHAPTER 3 Experimental design

end of the experiment. For instance, in the ATM experiment, if the touch-screen in-
terface is always tested after the button interface, we might draw a conclusion that the
touch-screen interface is not as effective as the button interface when the observed
difference is actually due to the participants' fatigue. We might fail to identify that
the touch-screen interface is better than the button interface because the impact of
fatigue offsets the gain of the touch-screen interface. Similarly, the potential problem
of fatigue can also be controlled through the adoption of the Latin Square Design.
3.3.1.3 Comparison of between-group and within-group designs
The pros and cons of the between- and within-group designs are summarized in
Table 3.1. You can see from the table that the advantages and limitations of the two
design methods are exactly opposite to each other.

Table 3.1 Advantages and Disadvantages of Between-Group Design and
Within-Group Design
Type of Experiment Design
Between-Group Design Within-Group Design
Cleaner Smaller sample size
Avoids learning effect Effective isolation of individual
Advantages
Better control of confounding differences
factors, such as fatigue More powerful tests
Larger sample size Hard to control learning effect
Large impact of individual Large impact of fatigue
Limitations differences
Harder to get statistically
significant results

3.3.2 CHOOSING THE APPROPRIATE DESIGN APPROACH
It is quite common for experimenters to argue back and forth when deciding which
of the two design approaches to adopt. Many times the decision is quite hard to make
since the advantages and disadvantages of the between-group design and within-group
design are exactly opposite to each other. It should be emphasized that each experiment
is unique and the decision should be made on a case-by-case basis with full consider-
ation of the specific context of the experiment. In some cases, a hybrid design may be
adopted that involves both between-group factors and within-group factors. The hy-
brid approach is discussed in detail in Section 3.4.2. This section discusses the general
guidelines that help us choose the appropriate approach for a specific user study.
3.3.2.1 Between-group design
Generally speaking, between-group design should be adopted when the experiment
investigates: simple tasks with limited individual differences; tasks that would be
greatly influenced by the learning effect; or problems that cannot be investigated
through a within-group design.