Page 78 - The Importance of Common Metrics for Advacing Social Science Theory and Research

The Importance of Common Metrics for Advancing Social Science Theory and Research: A Workshop Summary
http://www.nap.edu/catalog/13034.html

66 THE IMPORTANCE OF COMMON METRICS

compelling examples of the theoretical and practical implications of differ-
ent definitions of social mobility. For example, what to measure and the
appropriate metrics would be different for understanding and comparing
social status and relationships intergenerationally than if the purpose is to
monitor and promote equal opportunity in education or economic welfare.
In Maynard’s view, Hoyle made a convincing case that for self-
regulation there is neither a compelling need for a common metric nor is it
likely that there would ever be a need for a single measure. The concept of
self-regulation varies with age, with setting, and with goal. It is an umbrella
concept that, for scientific, political, and practical purposes, would prob-
ably need to be greatly refined and tailored to the intended use.
For Maynard, one of the implications from this meeting is that it would
be desirable to embark on a strategy of encouraging and facilitating the use
of common metrics in cases in which there are well-established, meaningful
metrics or when such measures could be constructed and made accessible
with reasonable effort. This could take the form of doing a better job of
ensuring that the good metrics are well defined, have established psycho-
metric properties, and that the means for application of these measures is in
the public domain. Royalties for the use of measures would be a deterrent
to adoption, regardless of their quality.
Maynard also shared three smaller observations

1. The process of developing common metrics will be facilitated by
encouraging the adoption of common items (anchor items) that can
provide cross-walks across studies that are using different measures
of similar constructs intentionally—for example, because their con-
texts or purposes differ or because they are still working on good
measure development.
2. Greater use of “linking” studies could and should be encouraged
when there is an interest in comparing across studies or data sets
using different measures of purportedly the same construct, like
poverty or social mobility.
3. It may be necessary to change the incentive structure for the sci-
entific community to discourage the creation of new measures for
the wrong reasons, such as to advance a professional career or
for financial gain. More thought needs to be given to rewarding
researchers for replicating and extending and to the relevance of
the measures and the metrics.

OPEN DISCUSSION
Sheila Jasanoff (Harvard University) began the discussion by asking
whether there is benefit to thinking about standardization itself as being

Copyright © National Academy of Sciences. All rights reserved.