Page 57 - Becoming Metric Wise

47
Publishing in Scientific Journals

one, four, or five reviewers. Note that some editors strongly favor two
reviewers and try to make a decision based on these two reports, while
others select three reviewers from the start. Considering three cases,
namely rejection when all reviewers recommend rejection, rejection when
any reviewer recommends rejection, and rejection when the majority
recommends rejection, he found that for this journal a manuscript was
mostly rejected once one reviewer recommended rejection.
As is clear from Schultz’ investigation one may also use three reviewers
from the start. Then one must wait for the results and decide by majority
rule. If one uses the clear-cut approach, then one just waits for one N or
three Y. Yet, instead of waiting for three reports, one may try to decide on
the basis of the two reports arriving first. Only when there is a difference
of opinion, does one wait for the third review and decide by majority.
Realizing that selection processes are never faultless, Bornmann and
Daniel (2009a) investigated the predictive validity of the manuscript selec-
tion process at Angewandte Chemie International Edition (AC-IE) and con-
ducted a citation analysis for 1817 manuscripts that were accepted by the
journal, or rejected but published elsewhere. They calculated the extent
of type I error (defined here as accepted manuscripts that did not perform
as well as or worse than the average rejected manuscript, hence their cita-
tion performance was overestimated) and type II errors (defined here as
rejected manuscripts that performed equal to or above the average
accepted manuscript for these submissions their citation performance was
underestimated) of the selection decisions. They found for both types of
errors 15% of wrong decisions. It should be observed that AC-IE is a
very selective journal which publishes manuscripts only if two external
referees consider the results of the study reported in the manuscript as sig-
nificant and recommend publication in AC-IE.
Making type II errors (rejecting “good” manuscripts) is a missed
opportunity for a journal. Moreover, setting the bar for acceptance too
high inevitably leads to the rejection of good, and even excellent or
breakthrough manuscripts.
Theoretically one may wonder if, assuming there were three referees,
and one makes a decision when two reviews are received, what would be
the difference when the third advice actually comes in second (and hence
the one that was second previously is not taken into account).
Concretely, Egghe (2010a), inspired by (Bornmann & Daniel, 2009b)
studied the following two situations. The first one is the so-called 50-50
rule in which the editor randomly makes a Y decision in 50% of the cases