Page 133 - Composition in Convergence The Impact of the New Media on Writing Assessment

100 CHAPTER 4
100
CHAPTER
4
and without closure, the deep assessment mechanisms used to study
these students' work must have great flexibility. In addition, any as-
sessment strategies for these contexts need to have some common
ground with the historical understanding of validity and reliability
to gain the respect of the quantitative folks who generally sit in deci-
sion-making capacities on campus and who frequently deride any
measurement system that does not look like a numeric study.
One method for building credibility in deep assessment is to de-
velop what is called in media research an analytic induction strat-
egy (Wimmer & Dominick, 1997). In this technique, the evaluator
forms a hybrid between quantitative and qualitative research
methods. The first step in this strategy is for the assessment team
or the instructor to state clearly the criteria to be investigated and
to construct a hypothesis to guide the evaluation procedure. Next,
the instructor pulls a representative sample from the entire group
of students using a commonly recognized random sampling for-
mula like "1 in X." For an instructor who teaches four sections of
composition (approximately 100 students), she can use her class
lists to select a 1 in 10 sample to pull 10 students' electronic as-
signments at random to study further. Then she can examine a
single case from the representative sample to test the hypothesis.
If problems occur in the evaluation, she reworks the hypothesis or
the criteria and tests again. Otherwise, she judges the remaining
cases from the representative sample, looking for patterns and
themes that refine her hypothesis. When finished, the teacher re-
turns to the 10 sample assignments to study any negative cases
that could disprove her hypothesis. If problems occur, the instruc-
tor again reworks the hypothesis and continues testing. Although
this is a time-consuming activity, the teacher should develop a
very strong argument that maintains elements of quantitative
and qualitative assessment. Moreover, this inductive and recur-
sive evaluation method has credence in other academic communi-
ties; numerous educational researchers, for instance, find this
form of naturalistic inquiry to be a valid form of accountability
(Hopkins, 1998). This becomes important if a faculty member or a
program must present harder data to administrators or faculty
senates to support curriculum matters.
A second, somewhat less labor-intensive, way of building a deep
assessment context that is credible without relying on historical un-
derstandings of validity and reliability also comes from media re-