Page 126 - Composition in Convergence The Impact of the New Media on Writing Assessment

VALIDITY AND RELIABILITY 93

lems in the assessment mechanism. The difficulty with this practice
is that an evaluation tool can have face validity and be completely in-
valid in the more important areas of tests and measurements. Of
course, when many compositionists realize this point—generally
when they are in some administrative or other across-the-campus
meeting—the results are sometimes less than amiable. Writing fac-
ulty need to be aware that problems of validity frequently occur in
assessment because writing is an indirect process, and, if educational
researchers are truthful about the subject, no one is totally certain
that what is being measured in a piece of writing is precisely what is
intended to be measured. This is not a flaw with the teaching faculty
or the students' achievements in the class; this is a condition of try-
ing to evaluate the unknowable—that is, how each person creates a
written product. After all, there are significant connections to craft
and to aesthetics in writing, and those variables cannot be measured
objectively and quantified.
How validity and reliability have been presented in writing assess-
ment, especially in writing assessment that is post-indirect method,
reflects the language of an earlier, psychometric understanding of
writing. For psychometricians, writing can be reduced to discrete
variables addressed by multiple choice quantifiers. So, it makes sense
that terms applicable to quantifiable research be used to describe
evaluation. Composition, writing instructors should hope, has
moved beyond this point. In the last 25 years, the field of Composi-
tion Studies has progressed in the direction of qualitative and action
research methodology in both its scholarship and assessment philos-
ophies. The advent of poststructuralism and postmodernism ush-
ered in social constructivism, ethnography, content analysis, and
discourse analysis—none of which paralleled the quantitative pro-
cesses. Consequently, changes have occurred in the ways in which
writing assessment is conducted. Holistic scoring, in principle, corre-
sponds to the qualitative researcher's belief that writing cannot be
divided into subparts and the entire work must be looked at as a
whole unit. However, to mollify the psychometricians, holistic scor-
ing in writing has numerous subsets and criteria that do indeed di-
vide the students' work into pieces. These subsets and criteria form
the rubrics that teachers use regularly in evaluating writing.
Similarly, portfolios also correspond to the qualitative position
that writing can be assessed only after students engage in an inten-
sive, lengthy involvement working with a series of texts. Instructors