Page 295 - Becoming Metric Wise

287
Research Evaluation

Shanghai (ARWU) Ranking. Of course when scientists receive such a
prize while they are still young this has a huge influence on their further
career.
Of much less notoriety we have the Derek J. de Solla Price Award for
scientometrics. A list of those colleagues who obtained this award is
included in the appendix.

8.13 FURTHER REMARKS

In this Section we bring together some further remarks and observations
related to different aspects of research evaluation.

8.13.1 Single Metrics Versus Summary Metrics
A single metric can mislead (and often will) and when it leads to decisions
about individuals or project, it may misdirect. When using a single met-
ric, all too often one ignores its shortcomings. This may lead to a distor-
tion of reality and to scientists paying only attention to what is measured.
Hence a whole battery of indicators, should be made available for
peer review, taking into account that one metric among other ones can
provide essential information. As a general rule one should discard as little
information as possible when performing evaluations.
Composite indicators may be useful as summary measures, but, in our
opinion, depend too much on the weights assigned to each component.
Values and rankings based on such composite metrics are difficult to
interpret and conceal trade-offs that should be made open.

8.13.2 Reputation

In a study on determinants of scholarly reputation in the field of manage-
ment Dewett and Denisi (2004) found that quality of research (as mea-
sured by the proportion of refereed articles in premier journals, editorial
activity and received awards), and in particular perceived creativity, is the
main determinant for reputation. When determining reputation this vari-
able is more important than sheer quantity of output.

8.13.3 Short-Term Versus Long-Term Objectives
Nowadays societal problems tend to favor short-term, goal-oriented pro-
jects over long-term basic research. “Scientists have to describe in advance all