Page 328 -

11.4 Analyzing text content 317

11.4.3.2 Reliability
The ambiguous data that is the focus of content analysis exemplifies many of the reli-
ability challenges presented by qualitative data analysis. The same word may have
different meanings in different contexts. Different terms or expressions may suggest
the same meaning. The data may be even more ambiguous when it comes to the inter-
pretation of body language, facial expression, gestures, or art work. The same people
may interpret the same gesture differently after viewing it at different times. In many
studies, the data set is very large and multiple coders may code different subsets of
the data. Due to the nature of content analysis, it is more vulnerable to biases and
inconsistencies than the traditional quantitative approach. Therefore, it is particularly
important to follow specific procedures during the coding process and use various
measures to evaluate the quality of the coding. The ultimate goal of reliability control
is to ensure that different people code the same text in the same way (Weber, 1990).
Reliability checks span two dimensions: stability and reproducibility. Stability is
also called intracoder reliability. It examines whether the same coder rates the data
in the same way throughout the coding process. In other words, if the coder is asked
to code the same data multiple times, is the coding consistent time after time? If the
coder produces codes that shows 50% in category A, 30% in category B, and 20%
in category C the first time; then 20% in category A, 20% in category B, and 60% in
category C the second time, the coding is inconsistent and the intracoder reliability
is very low.
In the context of content analysis, intercoder reliability is widely adopted to mea-
sure reproducibility. It examines whether different coders code the same data in a
consistent way. In other words, if two or more coders are asked to code the same data,
is their coding consistent? In this case, if one coder produces codes that shows 50%
in category A, 30% in category B, and 20% in category C; while the other coder pro-
duces codes that show 20% in category A, 20% in category B, and 60% in category
C, then the coding is inconsistent and the intercoder reliability is very low.
A further step in demonstrating reliability might use multiple coders specifically
chosen for differences in background or theoretical perspectives, leading to a theo-
retical triangulation (Stake, 1995). If individuals with substantially different intel-
lectual frameworks arrive at similar conclusions, those results may be seen as being
very reliable.
In order to achieve reliable coding both from the same coder and among multiple
coders, it is critical to develop a set of explicit coding instructions at the beginning of
the coding process. All of the coders need to be trained so that they fully understand
the instructions and every single coding item. The coders then test code some data.
The coded data is examined and reliability measures are calculated. If the desired
reliability level is achieved, the coders can start the formal coding. If the desired
reliability level is not achieved, measures must be taken to improve reliability. These
measures might include retraining and recoding the data used in the test coding.
Alternatively, the coders might use a discussion of disagreements to determine how
coding should be conducted, and revise the codebook and coding instructions to
reflect the new consensus. After the formal coding process starts, it is important to