Page 146 - Composition in Convergence The Impact of the New Media on Writing Assessment

VALIDITY AND RELIABILITY 113

iar with electronic communication can distinguish acceptable work in
ways that break from the linear holistic scoring guides, yet still retain
the sense of reliability that many test-and-measurement people want
to see in outcomes assessment.
This "new" reliability does not depend on the consistency of writ-
ing specialists guessing the same score to keep consensus and
interrater reliability or to ensure the reliability of test instrument,
two situations that frequently lead to a Panopticon of sorts in assess-
ment settings. Instead, this new reliability insists on faculty review-
ers who are experienced with the currency of technological conver-
gence and student e-texts to make decisions about the adequacy of
students' writing in these genres.
Assessing for adequacy moves deep assessment closer to validity
because the evaluators have the opportunity to examine a fuller
scope of the students' writing activities and contexts. Not only will
the archived data contain numerous examples of writing produced
under various conditions and for various audiences, the students'
own analytical examinations of their work and the instructors'
points of intervention; the data should also reflect the teacher's
comments. All these elements provide the breadth needed to make a
valid writing assessment. Deciding whether a student's archived
writing is acceptable depending on local criteria should pass the test
for face validity because the data are evaluated by local experts using
familiar criteria to measure the writing. Additionally, when a panel
or team of teachers who are experienced in electronic communica-
tion evaluates the students' adequacy as writers of online material,
there is also predictive validity. That suggests that evaluators ac-
knowledge the students are reasonably able to do the work again
later based on examining the archived materials. Concurrent valid-
ity can be included in this type of evaluation if the assessment team
wants to measure the students' electronic writing against students'
F2F writing; this approach might be an especially useful step in writ-
ing programs where there are computer-only sections (and, con-
versely, F2F-only sections). Testing for concurrent validity will be
useful only if the criteria used to measure the writing remain identi-
cal for both sections. It is also important to note that in programs
where computer-enhanced composition is under fire or where there
is great skepticism, testing for concurrent validity may answer
administrators' or faculty members' concerns about the benefits of
using computer technology in the writing classroom.