Page 293 - Becoming Metric Wise

285
Research Evaluation

bad habit, especially when large groups reach a consensus. Yet, such an
approach also entails some problems. Even in the exact sciences, where
there is a large consensus about the list of journals included in the Journal
Citation Reports some problems may occur.
• how to compare related subdisciplines?
• what is the relation between a journal and the articles included in it?
It has been shown (Seglen, 1989, 1992, 1994, 1997; Milojevi´ c, et al.,
2017) that there is only a weak correlation between the impact factor of a
journal (basically a measure for the visibility or popularity of a journal)
and the actual impact of an article published in this journal. Yet, among
factors determining impact the JIF 2 seems to be the one with the highest
correlation (Bornmann & Leydesdorff, 2017). The main reason for this is
the skewness of citation results, which implies that a few heavily cited
articles determine the average i.e., the impact factor (Rousseau, 2014b).
Lowly cited articles—the majority—are then “rewarded” for the citations
received by a few other ones. This may happen to articles dealing with
“fancy” topics which are easily accepted but are not necessarily of high
quality and may not receive a lot of citations. Drawing lists of “top jour-
nals” for a field is often done in the social sciences or the humanities and,
it seems, particularly often in business schools, for which Steward and
Lewis (2010) report that nearly 40% used an internally developed list. Van
Fleet et al. (2000) provide a table of advantages and disadvantages of
drawing such lists.
Disadvantages
• Drawing a list is often difficult and it may take a long time before a
consensus is reached.
• It may lead to stereotypical publication behavior.
• It may harm scientists’ careers if such lists are very local.
• May harm colleagues active in a small, specialized domain.
• It discourages launching new journals.
• It provides more power to editors whose journal is included in the
list.
Advantages
• It provides explicit targets.
• It reduces the time needed to perform evaluations.
• Such lists are useful for benchmarking.
An important practical problem is that scientists, especially in the
social sciences and the humanities, have a tendency to consider those
(local) journals used by themselves as those of importance. This makes