Page 335 -

324 CHAPTER 11 Analyzing qualitative data

19. What can you do to improve the validity of the findings of a HCI study?
20. Why do you need to conduct reliability checking during and after the coding
process?
21. What is “stability” in the context of a reliability check?
22. What is “reproducibility” in the context of a reliability check?
23. What is the formula for computing Cohen's Kappa?
24. How do you interpret a specific value of Cohen's Kappa?

25. What is the difference between intracoder reliability and intercoder
reliability?
26. What is the advantage and disadvantage of using a subjective coder?

27. What is the advantage and disadvantage of using an objective coder?
28. Why is analyzing multimedia content difficult?
29. How does the partially automated annotation method work?

RESEARCH DESIGN EXERCISE

You interview 50 children between the ages of 8 and 15 to study their computer usage
behavior. During the data analysis, you find that the objective of using computers can
be grouped into three categories: educational, communication, and entertainment.
Two coders independently code the data and the agreement of their coding regarding
computer usage objective is summarized in Table 11.5. Answer the following ques-
tions based on the agreement table:
1. Develop an agreement matrix. (Hint: You need to compute marginal
totals for each row and column and the expected percentage agreement
for each cell.)
2. Calculate Cohen's Kappa.
3. Discuss the result and determine whether the coding is reliable.

Table 11.5 Children's Computer Usage Objectives Coding Agreement
Coder 2
Education Communication Entertainment
Education 0.49 0.05 0.02
Coder 1 Communication 0.03 0.11 0.01
Entertainment 0.04 0.02 0.23