Page 261 - Becoming Metric Wise

253
Research Evaluation

No Database has Complete Coverage
Reliable conclusions about the work of a scientist or group of scientists can
only be deduced if one has a good view on all scientific results of the
research unit. Yet, the big international bibliographic databases do not have
as their aim to cover all publications, let alone other scientific output.

Publication and Citation Windows Must Depend on the Field Being
Evaluated
Using an adequate publication and/or citation window is of the utmost
importance. Such windows differ between research fields. Moreover,
some scientific domains depend on external circumstances on which they
have no influence at all. An interesting case is the field of zero-gravity
experiments. Researchers active in this field depend on space crafts.
Room for their instruments must be reserved/negotiated years in advance;
there are often delays, and sometimes failures (Challenger, Columbia). It is
clear that these researchers cannot be evaluated using the same short-term
windows as colleagues who perform experiments on earth (Nederhof
et al., 2012).

Differences Between Scientific Disciplines
Not only time, in the case of publication and citation windows, plays a
role when comparing different disciplines. Scientific domains often differ
in the way reference lists are drawn. In some fields it is customary to
include a short or sometimes long literature review, while in other fields
one mainly cites what is actually used. Hence, it makes sense to normalize
indicators based on disciplines. This is not a sinecure as even a small field
such as library and information science has differences in citation poten-
tial: In recent years, articles dealing with networks (in LIS) or altmetric
issues receive more citations than articles on librarianship. As a rule of
thumb it may be said that it is best not to compare scientists active in dif-
ferent fields (why would one?). However, the question then becomes
how fields and subfields should be delineated.
A balanced discussion on normalization is provided in Ioannidis et al.
(2016). It covers normalization with respect to scientific fields, year of
publication, document type and database coverage. The authors mention
that all in all normalization is performed to correct for the imbalance of
citation opportunity.