Page 260 - Becoming Metric Wise

252 Becoming Metric-Wise

8.1.5 Criticism on Using Citations for Research Evaluations
Although indicators as presented in the previous sections and chapters are
used on a large scale, not everyone is convinced about their validity. Let
us say that we admit that each of these criticisms contains some truth, but
it is our opinion that careful peers can, nevertheless, make sure that
citation-based indicators are used during research evaluation exercises in a
sensible way.
Schneider (2009), using among others work by Gla ¨ser and Laudel
(2007), provides the following list of reasons why, according to him, cita-
tions can never lead to a valuable and generally useful measure for
research quality. We added some comments to his list.

Citations do not Measure the Quality of an Article or a Set of Articles
Citations do not lead to a measure of quality but of visibility. Visibility
depends on quality, but also on many other factors such as the publication
language, the type of publication (article, review, contribution in an edi-
ted book, conference contribution), the scientific discipline, the publisher
and its website, the prestige of author(s) or institute, the topicality of the
contribution and pure luck.

Citation Statistics, Especially for Single Articles, do not Always Reflect
Scientific Excellence
In order for scientometric indicators to yield reliable results (where we
use “reliable” in the sense of “comparable with other ones”) a large
enough set of publications must be available. Sometimes a publication
may receive quite a number of citations or sometimes it is barely cited,
but little can be decided from this on statistical grounds. Only peer
review may—perhaps—find out that an article which received just a few
citations is the far higher intellectual achievement. This happened for
instance with Nobel Prize winner Youyou Tu’s article published in the
Chinese Medical Journal (Tu, 1999). This is just one of the four articles
she published which is included in the Web of Science. By the end of
2015 this article, strongly related to her Nobel Prize, had received (only)
19 citations. Yet, 8 articles citing (Tu, 1999) had already received more
than 19 citations; the most cited one even more than 100. It is highly
unlikely that these citing articles include more highly intellectual content
than Tu’s. Articles such as Tu’s are referred to as under-cited influential
articles (Hu & Rousseau, 2016, 2017).