Page 125 - Composition in Convergence The Impact of the New Media on Writing Assessment

92 CHAPTER 4

Certain types of assignments—such as argumentative writing—
create more errors in students' writing because of the increased de-
mands the activity places on the thinking and information-gather-
ing processes and because ecological or individualistic fallacies that
teachers have about writing interfere with readers' abilities to
make decisions on many of these pieces. 2
Equivalence frequently causes problems in writing assessment,
which affects an assessment's reliability. As Lauer and Asher (1988)
described equivalency reliability, two sets of test scores given simul-
taneously to a sampling of people are correlated to each other. All as-
pects of the data must be the same (equivalent), such as the averages,
standard deviations, and average intercorrelations among items be-
fore running a correlation of the data. Although equivalence works
well with a standardized, indirect writing test, the problem arises
with holistic readings of essays and portfolios, because the data
across two assessment settings may not be equivalent based on vari-
able error. It is entirely possible to have differing averages, standard
deviations, and average intercorrelations among items that can
skew correlations.
Validity is also a problem for writing mechanisms because of errors
caused by the lack of face validity. This is simple error in that the test
does not measure what it appears to measure on the surface of the test.
But other problems for validity exist as well. The inability of a testing
instrument—whether testing indirect or direct writing—to predict a
student's future success in writing can also affect some forms of valid-
ity. So can unstated criteria used to pretest the assessment mechanism
affect some aspects of validity. Moreover, validity can also be affected
by the lack of connecting an essay, portfolio, or multiple-choice exam
to any departmental or pedagogical framework.
Too often writing faculty and their departments go for the easy
form of validity, face validity, as their defense against other prob-

In empirical research, an ecological fallacy refers to using aggregate data that help to
analyze a group to make inferences on the behavior or properties of an individual or indi-
viduals. For example, in a holistic reading (portfolio or essay structure), applying gender
or racial statistical data for the campus or the region to assess an individual student's
work would be using an ecological fallacy. The opposite of an ecological fallacy is the indi-
vidualistic fallacy. In this situation, a reader makes inferences about an entire group of stu-
dents or an educational system in general on the basis of a single student's work. An
example of this would be condemning all high school writing instruction on the basis of
one student's writing sample. See Nachmias and Nachmias (1981, p. 57) for a social scien-
tist's perspective on these two fallacies.