Page 60 -

CHAPTER
3
Experimental design

Experiments help us answer questions and identify causal relationships. Well-
designed experiments can reveal important scientific findings. By contrast, ill-
designed experiments may generate results that are false or misleading. Experiments
have been widely used in the human-computer interaction (HCI) field to develop and
modify user models or task models, evaluate different design solutions, and answer
various other critical questions, such as technology adoption.
Before we discuss specific experimental design methods, we need to differen-
tiate three groups of studies: experiments, quasi-experiments, and nonexperiments
(Cooper and Schindler, 2000; Rosenthal and Rosnow, 2008). Figure 3.1 demon-
strates the relationship among the three types of studies. If a study involves multiple
groups or conditions and the participants are randomly assigned to each condition,
it is a true experiment. If a study involves multiple groups or conditions but the par-
ticipants are not randomly assigned to different conditions, it is a quasi-experiment.
Finally, if there is only one observation group or only one condition involved, it is a
nonexperiment. True experiments possess the following characteristics:
• A true experiment is based on at least one testable research hypothesis and aims
to validate it.
• There are usually at least two conditions (a treatment condition and a control
condition) or groups (a treatment group and a control group).
• The dependent variables are normally measured through quantitative
measurements.
• The results are analyzed through various statistical significance tests.
• A true experiment should be designed and conducted with the goal of removing
potential biases.
• A true experiment should be replicable with different participant samples, at
different times, in different locations, and by different experimenters.
In this chapter, we focus on the design of true experiments, which means that all
the studies we discuss have multiple conditions or measures and the participants are
randomly assigned to different conditions. We start with the issues that need to be
considered when designing experiments, followed by discussions of simple experi-
ments that involve only one independent variable. We then examine more compli-
cated experiments that involve two or more independent variables. Three major types
of experiment design are discussed: between-group design, within-group design,
and split-plot design. Section 3.5 focuses on potential sources of systematic errors

Research Methods in Human-Computer Interaction. http://dx.doi.org/10.1016/B978-0-12-805390-4.00003-0 45
© 2017 Elsevier Inc. All rights reserved.