Page 134 - Composition in Convergence The Impact of the New Media on Writing Assessment

VALIDITY AND RELIABILITY 101

search. Maykut and Morehouse (1994) proposed four conditions for
establishing trustworthiness in reporting qualitative data. Here I
have adapted Maykut and Morehouse's 1994 model for use in a
writing assessment situation:

1. Collect data from numerous sources. Student e-texts, protocols,
e-mail exchanges, listserv archives, and other classroom artifacts
show others that the evaluator studied the students' work from
many perspectives. This content does not necessarily have to be in an
electronic portfolio; in fact, a polished e-portfolio may hinder the
study. An electronic portfolio does not provide the raw information
needed for the evaluator to truly measure growth and change in the
student writers' processes. A few "finished" pieces might be useful,
but instructors should have a mix of work in various stages, genres,
and contexts for more believability in the evaluation.
2. Develop an audit trail. Whether in a LAN space, or on a disk,
CD, or keychain hard drive, teachers need to create a safe, perma-
nent record of the class's original data, comments about the data,
and any analytical methods used to conduct the assessment. This
permanent record is called an "audit trail." Audit trails are the eas-
iest way to demonstrate the possibility of replication in qualita-
tive research. Audit trails are especially important when arguing
the validity of an assessment procedure, as anyone can retrace an
instructor's steps and check the results for accuracy. An audit
trail for 10 students may take up to several kilobytes in a single
assessment, though, so instructors might want to consider putt-
ing this information on some removable, easily stored, perma-
nent system (rewriteable CD-ROM or DVD, for instance) if they do
not have access to their campus network or if the bandwidth ex-
ceeds campus allocations.
3. Conduct member checks. As teachers make notes and develop
conclusions about what they read, those on the assessment team
review their findings with each other and with the students to en-
sure accuracy in reporting the material. This step is very impor-
tant if instructors use protocol interviews with students, because
teachers will want to be certain that they precisely capture the
students' words and intentions.
4. Develop an assessment team to avoid epoche. In research speak,
epoche means researcher bias or prejudice. Establishing a team of
evaluators to keep everyone focused on the criteria when describing