Page 232 -

220 CHAPTER 8 Interviews and focus groups

Effective analysis works to avoid bias and reliance on preconceived notions. The
absence of “hard,” numeric data makes interview responses and similar qualitative
data sources particularly susceptible to biased manipulation. An emphasis on data
points that confirm your favorite hypothesis, at the expense of comments that argue
against it—a practice known as “cherry picking”—is just one of the possible biases
in the analysis of results. Biased consideration of responses from specific partici-
pants, or classes of participants, can be a problem for focus group data. If your analy-
sis pays disproportionate attention to female participants relative to male participants
(or vice versa), any resulting interpretation will be somewhat distorted. Your analysis
activities should always strive to be inclusive and data driven.
Additional information about the use of qualitative data analysis methods can be
found in Chapter 11.

8.10.1 WHAT TO ANALYZE
Fully structured interviews consisting only of closed questions are the easiest to ana-
lyze. As all interviewees are asked the same questions and all answers are taken from
a small set of possibilities, analysis is essentially a tabulation problem. You can tabu-
late the frequency of each answer and use straightforward statistical tests to deter-
mine when differences in response rates are meaningful (see Chapter 4). Quantitative
results can also be used to group characteristics (see the Finding and Reminding
sidebar in Section 8.3.1 for an example).
Analysis gets harder as your questions become more open-ended and the interview
becomes less structured. Open-ended questions can be answered in a different way by
each interviewee. Two participants might answer any given question in entirely different
ways, creating the challenge of identifying the common ground. Unstructured or semis-
tructured interviews introduce the additional complication of questions and topics arising
at very different stages in different interviews. Analysis of these interviews may require
tying together comments made at very different times under very different contexts.
Should your analysis be based on written notes or on audio or video recordings?
Unlike written notes, recordings provide complete and unfiltered access to every-
thing that an interviewee said or did, even months after the fact. This record can be
used to reconstruct details, focus in on specific comments, and share user feedback
with colleagues. The disadvantage, of course, is the expense and challenge of wad-
ing through hours of video or audio data. You can analyze recordings by listening to
comments piece by piece, repeatedly replaying pieces of interest until you gain an
understanding, but this can be a slow, often tedious process. Verbatim transcriptions
translate these hours of discussions into pages of written text that might be more
amenable to analysis and editing via software, but transcribing can also be an expen-
sive and unappealing process. Although it may be possible to use automated speech
recognition techniques to generate a transcript, these tools are subject to recognition
errors that might limit the quality of the output.
Notes written during the interview have the advantage of being relatively compact
and easy to work with. Your written notes may omit some interesting details, but it's
likely that the comments you managed to get down on paper were among the most