Page 200 - Becoming Metric Wise

190 Becoming Metric-Wise

method to rank journals. Generally speaking, existing ranking methods
can be classified as peer review based, citation (or indicator) based or a
combination of the two.

6.18.1 Peer Review Based Approach
In this approach, the assessment of scientific journals in a particular field or
subfield is undertaken by a group of scholars working in the same area. In
most cases a survey or questionnaire is designed to collect opinions from
professors or academic administrators (Chandy et al., 1991). The peer
review based approach is expected to produce a valid assessment because
experts (not just any scientist) judge the quality of a journal based on their
experience and accumulated impressions of the quality of the published
papers that they have read or reviewed. However, it is unclear on what cri-
teria experts base their judgments. Thus, it is not clear how or even
whether assumed quality criteria are measured in any of these rankings.
More to the point, these quality criteria are rarely made explicit nor are
their relative weights in the overall assessment. One may say that journal
rankings based on peer review are reputation rankings while the citation
based approach, discussed next is a visibility or influence based approach
(Christenson & Sigelman, 1985). Christenson and Sigelman also found that
scholars in sociology and political sciences tended to establish reputations
that stick and are not re-evaluated in light of recent publications.
A comprehensive list of the advantages and disadvantages of the expert
survey ranking method is provided in Table I of Serenko and Bontis
(2013), which we reproduce here with some adaptations.
Advantages of expert survey journal ranking:
a. Suitable for rankings on all levels, in particular for local and
regional journals.
b. Suitable for rankings in new disciplines.
c. New journals can be included.
d. This approach is widely accepted.
e. It reflects the multifaceted opinion of a representative group.
f. It is difficult to manipulate by outsiders.
Disadvantages of expert survey journal ranking:
a. Subjectivity as experts may be influenced by their personal research
interests and those of leading figures.
b. Familiarity bias: respondents assign higher values to journals they
are familiar with.