Page 25 - Build a Culture of Employee Engagement with the Principles

xxiv Introduction

The instrument had been purchased several years earlier
from a psychologist who assured Mary and John that it was valid
and reliable, although he did not provide any documentation. I
took the test, read the computer-generated report, and sensed
immediately that it was a bad instrument. With the raw data
of several thousand completed reports, it did not take me long
to confirm my suspicion. The instrument failed even the most
basic tests of reliability and validity. In fact, some of the scales
and the manner in which they were scored made no sense at all.
Had I still been teaching statistics and survey development, this
would have served as an ideal example of what not to do. The
report, which was generated and used to make decisions about
people’s careers, had all of the validity of a fortune cookie.
I explained my findings to Mary and John as straightfor-
wardly as possible and let them know that they needed to
immediately stop using the instrument. Put simply, they were
committing fraud. John and Mary listened without comment
and then asked me to step outside the door. After a few minutes
Mary called me back in and said, “I think that pulling the instru-
ment would confuse our clients.” I was speechless. Mary asked
if I could revise it—she liked this idea because they could then
market it as a new and improved version. I told her that it might
be possible to create a similar-looking instrument but that the
majority of existing items would have to be thrown out. I also
told her that the development and validation process would take
several months.
I spent the next six weeks creating, testing, editing, and
retesting items. After a dozen different versions I met with John
and Mary to let them know that we were ready to begin the pilot
study. As part of the research plan, four hundred employees
from their largest client were to be surveyed. Mary praised me
for a job well done and told me that plans had changed: there
would be no pilot study. She had confidence in me and the new