Page 48 - Becoming Metric Wise

38 Becoming Metric-Wise

distributing grant money based on proposals for scientific research. Most
aspects of peer review discussed here also apply to these other situations,
but this will not be the focus of this chapter. Peer review in research eva-
luations will be discussed in Chapter 8, Research Evaluation. In research
evaluation by peer review, experts may be supported by citation-based
evidence, but that is not possible for journal submissions. Peer review is
the only possible way to evaluate a submission.
Peer review for journal submissions has two aspects: A summative
one, leading to acceptance or rejection, and a formative one, leading to
improvements of the original submission. Note though, that the role of
reviewers is an advisory one and the editor-in-chief (EIC) is not obliged
to follow their advice.
As peer review is performed by humans it is subjective and opinions
about the content of a submission may differ. Indeed, reviewers some-
times show confirmation bias: this is the tendency to judge in favor of
what is already known and against submissions that contradict a reviewer’s
theoretical perspective (Nickerson, 1998). Especially highly innovative
research may encounter severe criticism. It is no surprise that many
Nobel Prize winners can tell stories about the rejection of their prize-
winning work (Campanario, 2009; Campanario & Acedo, 2007). Yet,
assessment of research of potentially ground-breaking developments is, by
its very nature, not straightforward. Authors claiming such breakthroughs
should provide sufficient evidence, while editors should facilitate scientific
debate on such claims.
Although the peer review system has a lot of problems, finding
solutions is very difficult, see e.g., Hochberg et al. (2009) and a comment
by de Mesnard (2010).
Complaints about peer review occur frequently e.g., see Santini (2005)
for a very entertaining article in which fake reviews on famous computer
science articles (Dijkstra, Codd, Turing, Shannon, Hoare, Rivest-Shamir-
Adelman) and a fake performance review of Einstein (www.norvig.com/
performance-review.html) are presented. In a systematic review on peer
review in biomedical journals, Jefferson et al. (2002) concluded that editorial
peer review, although widely used, is largely untested and its effects are
uncertain. Yet, already 3 years earlier van Rooyen et al. (1999) had devel-
oped the Review Quality Instrument (RQI) to evaluate review reports.
Since then, more studies of peer review, editorial and otherwise, and its
effects have been published e.g., see Bornmann and Daniel (2005, 2009b),
Langfeldt (2006), Sandstro ¨mand Ha ¨llsten (2008), and Mulligan et al. (2013).