Page 54 -

2.5 Limitations of experimental research 39

large sample size so that the difference can be observed even when the effect size is
relatively small. If interested, you can find more detailed discussions on statistical
power in Rosenthal and Rosnow (2008).

2.5 LIMITATIONS OF EXPERIMENTAL RESEARCH

Experimental research methods originated from behavioral research and are largely
rooted in the field of psychology. Experimental research has been a highly effective
research approach and has led to many groundbreaking findings in behavioral sci-
ence in the 20th century. Experimental research certainly plays an important role in
the field of HCI. A large number of studies that explored fundamental interaction
theories and models, such as Fitts' law, employed the approach of experimental re-
search. To date, experimental research remains one of the most effective approaches
to making findings that can be generalized to larger populations.
On the other hand, experimental research also has notable limitations. It requires
well-defined, testable hypotheses that consist of a limited number of dependent and
independent variables. However, many problems that HCI researchers or practitio-
ners face are not clearly defined or involve a large number of potentially influential
factors. As a result, it is often very hard to construct a well-defined and testable hy-
pothesis. This is especially true when studying an innovative interaction technique or
a new user population and in the early development stage of a product.
Experimental research also requires strict control of factors that may influence
the dependent variables. That is, except the independent variables, any factor that
may have an impact on the dependent variables, often called potential confound-
ing variables, needs to be kept the same under different experiment conditions.
This requirement can hardly be satisfied in many HCI studies. For example, when
studying how older users and young users interact with computer-related devices,
there are many factors besides age that are different between the two age groups,
such as educational and knowledge background, computer experience, frequency
of use, living conditions, and so on. If an experiment is conducted to study the two
age groups, those factors will become confounding factors and may have a sig-
nificant impact on the observed results. This problem can be partially addressed
in the data collection and data analysis stages. In the data collection stage, extra
caution should be taken when there are known confounding factors. Increasing the
sample size may reduce the impact of the confounding factors. When recruiting
participants, prescreening should be conducted to make the participants in differ-
ent groups as homogeneous as possible. When confounding factors are inevitable,
specific data analysis methods can be applied so that the impact of the confound-
ing factors can be filtered out. A common method for this purpose is the analysis
of covariables.
Lab-based experiments may not be a good representation of users' typical in-
teraction behavior. It has been reported that participants may behave differently in
lab-based experiments due to the stress of being observed, the different environment,