Page 76 -

3.5 Reliability of experimental results 61

60 Random errors only
Number of words entered per minute 50
Random errors plus bias
55

45

40

35

30
1 2 3 4 5
FIGURE 3.7
Comparison of random and systematic errors.

when biases are inevitable, and we need to isolate the impact of them from the main
effect when analyzing the data. There are five major sources of systematic error:
• measurement instruments;
• experimental procedures;
• participants;
• experimenter behavior; and
• experimental environment.
3.5.2.1 Bias caused by measurement instruments
When the measurement instruments used are not appropriate, not accurate, or not con-
figured correctly, they may introduce systematic errors. For instance, when observing
participants searching for an item on an e-commerce website, we may use a stop watch
to measure the time it takes to locate the specific item. If the stop watch is slow and
misses 5 minutes in every hour, then we consistently record less time than the actual
time used. As a consequence, the observed performance will be better than the actual
value. In order to control biases introduced by the measurement instruments, we need
to carefully examine the instruments used before experiment sessions. Another ap-
proach is to use extensively tested, reliable, and software-driven instruments. A bonus
of software-driven instruments is that they can avoid human errors as well.
3.5.2.2 Bias caused by experimental procedures
Inappropriate or unclear experimental procedures may introduce biases. As discussed
previously, if the order of task conditions is not randomized in an experiment with a
within-group design, the observed results will be subject to the impact of the learning
effect and fatigue: conditions tested later may be consistently better than conditions
tested earlier due to learning effect; on the other hand, conditions tested earlier may
be consistently better than later conditions due to fatigue. The biases caused by the