Page 56 -

2.6 Summary 41

performance and preference of the target population. Controlled experiments
have allowed us to make critical and generalizable findings that other
methods would not be able to provide. The truth is, experimental research and
significance testing is the only approach that enables us to make judgments with
systematically measured confidence and reliability. The control of confounding
factors is challenging but the impact of those factors can be reduced to
acceptable levels through well-designed and implemented experiments, which
we discuss in detail in Chapter 3.

2.6 SUMMARY
Research in HCI examines human behavior in relation to computers or computer-
related devices. There are three major types of research methods for studying human
behavior: descriptive, relational, and experimental. The major strength of experimen-
tal research, compared to the other two types, is that it allows the identification of
causal relationships between entities or events.
After a hypothesis is constructed, the design of an experiment consists of three
components: treatments, units, and the assignment method. In an experiment, the
process of sample selection needs to be randomized or counter-balanced, as does the
assignment of treatments, or experiment conditions. Many methods can be used to
randomly select samples or assign experiment conditions, including, but not limited
to, the random digit table and software-generated randomization schemes.
Successful experimental research depends on well-defined research hypotheses
that specify the dependent variables to be observed and the independent variables to
be controlled. Usually a pair of null and alternative hypotheses is proposed and the
goal of the experiment is to test whether the null hypothesis can be rejected or the
alternative hypothesis can be accepted. Good research hypotheses should have a rea-
sonable scope that can be tested within an experiment; clearly defined independent
variables that can be strictly controlled; and clearly defined dependent variables that
can be accurately measured.
Significance testing allows us to judge whether the observed group means are truly
different. All significance tests are subject to two types of error. Type I errors refer to
the situation in which the null hypothesis is mistakenly rejected when it is actually
true. Type II errors refer to the situation of not rejecting the null hypothesis when it
is actually false. It is generally believed that Type I errors are worse than Type II er-
rors, therefore the alpha threshold that determines the probability of making Type I
errors should be kept low. The widely accepted alpha threshold is 0.05. With its notable
strengths, experimental research also has notable limitations when applied in the field
of HCI: difficulty in identifying a testable hypothesis, difficulty in controlling poten-
tial confounding factors, and changes in observed behavior as compared to behavior
in a more realistic setting. Therefore, experimental research methods should only be
adopted when appropriate.