Page 16 - The Importance of Common Metrics for Advacing Social Science Theory and Research

The Importance of Common Metrics for Advancing Social Science Theory and Research: A Workshop Summary
http://www.nap.edu/catalog/13034.html

4 THE IMPORTANCE OF COMMON METRICS

fectively encouraged by grant-making institutions as part of the peer review
process and by journal editors.
Despite the interest in common metrics, some measures appear to defy
standardization. As we have come to understand race in social and cultural
terms, for example, the concept of race has become inherently difficult to
measure, let alone in a standardized way. Measures may obscure important
information in the underlying data or may fail to recognize the complexity
of the dimension of interest. In such cases, data in their raw, disaggregated
form are often more useful than when clothed in a composite measure or
in meta-analysis. Both location and metric can affect comparisons; calibrat-
ing individual scales, such as with the use of anchoring vignettes, can help
circumvent some of these problems rather than assuming a common scale.
Measures may also need to change over time, because concepts and
what society considers important change. For example, the concept of pov-
erty has changed over time, along with prices, products, and social norms—
and a useful measure will reflect these changes. In health care, ignoring
improvements in treatment would underestimate growth in medical output.
And in recent years, there has been greater interest worldwide in measuring
less tangible concepts, such as subjective well-being, satisfaction, and social
connectedness, as well as a movement from single measures to indices and
from activities to outputs and outcomes. And even if change is warranted,
changing a well-established measure may be difficult, if not impossible.
Although the exploration of common metrics is to be encouraged, the
meeting did sound a cautionary note on the prospects for useful and valid
common metrics in the social sciences and the dangers of using imperfect or
incomplete standardized measures to guide policy. Yet under certain situa-
tions, even an imperfect indicator can be good enough for promoting com-
petent discussion about actions to take. However, concerns were expressed
about the premature application of standards and the lack of appreciation
for the role of successful science in generating standardization. Participants
also noted that there is a risk that unnecessary standardization can mean
that weaknesses get codified and reinforced over time and that distortions
will occur from linking indicators too closely to policy decisions, particu-
larly if indicators are meant to promote accountability. Common measures
may also be lacking if there is no common understanding as to what the
measures represent.
Although theory is useful in the development of metrics, some common
metrics are not based on theory. An example is the unemployment rate, for
which no economic theory appears to apply.
Finally, measurement breakthroughs can take a long time and require
persistence, but the effort is well worth the investment. The development
of standard metrics that are useful in theory and in practice is important
and scientifically rewarding.

Copyright © National Academy of Sciences. All rights reserved.