Page 52 -

2.4 Significance tests 37

the null hypothesis when it is false and should be rejected (Rosenthal and Rosnow,
2008). A widely used example to demonstrate Type I and Type II errors is the judicial
case. In the US justice system, a defendant is presumed innocent. This presumption
leads to the following null and alternative hypotheses:
• H 0 : The defendant is innocent.
• H 1 : The defendant is guilty.
A Type I error occurs when the jury decides that the defendant is guilty when he
is actually innocent, meaning that the null hypothesis is rejected when it is true. A
Type II error occurs when the jury decides that the defendant is innocent when he is
actually guilty, meaning that the null hypothesis is accepted when it is false. Table 2.4
illustrates these errors. In the ideal case, the jury should always reach the decision that
the defendant is guilty when he is actually guilty and vice versa. But in reality, the jury
makes mistakes occasionally. Each type of error has costs. When a Type I error occurs,
an innocent person would be sent to prison or may even lose his life; when a Type II
error occurs, a criminal is set free and may commit another crime.

Table 2.4 Type I and Type II Errors in the Judicial Case
Jury Decision
Not Guilty Guilty
Not guilty √ Type I error
Reality
Guilty Type II error √

Let us further examine Type I and Type II errors through a study in the HCI do-
main. Suppose a bank hires several HCI researchers to evaluate whether ATMs with
a touch-screen interface are easier to use than the ATMs with buttons that the bank
branches are currently using. In this case, the null hypothesis and the alternative
hypothesis are:
• H 0 : There is no difference between the ease of use of ATMs with touch screens
and ATMs with buttons.
• H 1 : ATMs with touch screens are easier to use than ATMs with buttons.
The possible Type I and Type II errors in this study are illustrated in Table 2.5. A
Type I error occurs when the research team decides that touch-screen ATMs are easier
to use than ATMs with buttons, when they are actually not. A Type II error occurs
when the research team decides that touch-screen ATMs are no better than ATMs with
buttons, when they are. Again, each type of error can induce negative consequences.
When a Type I error occurs, the bank may spend money to switch to touch-screen
ATMs that do not provide better service to the customers. When a Type II error occurs,
the bank chooses to stay with ATMs with buttons and loses the opportunity to improve
the service that it provides to its customers.