Page 28 - The Importance of Common Metrics for Advacing Social Science Theory and Research

The Importance of Common Metrics for Advancing Social Science Theory and Research: A Workshop Summary
http://www.nap.edu/catalog/13034.html

16 THE IMPORTANCE OF COMMON METRICS

In the field of demography, the use of common measures is fairly well
accepted in the design and development of omnibus surveys. Although
this has many positive benefits, Bachrach noted that it also leads to the
development of “habitual measurement practices,” that is, relying on the
same measures regardless of whether they truly represent the theoretical
constructs of interest. For example, years of schooling are measured quite
similarly across the social sciences, although the measure is used to opera-
tionalize very different theoretical constructs ranging from opportunity cost
to human capital to social class. She echoed a point made by Hauser about
users reading into measures what they want. Thus, common measures alone
are insufficient if there is a lack of common understanding as to what those
measures represent. She identified the structure of peer review as yet an-
other set of factors that influences the health of measurement science in the
social sciences. NIH has recently shifted its review criteria to try to nudge
reviewers away from a very detailed focus on the technical approach used
in grant applications to a focus on impact, significance, and innovation.
There always has been tremendous variation across different review groups
as to how much attention is given to the quality of measurement and the
approach taken to measurement; she supposed that this new change may
further dampen attention to measurement. Bachrach saw similar variations
in the peer review of journal articles in terms of the importance accorded
to measurement issues.
Second, what is meant by common metrics? Bachrach encountered
multiple meanings in her reading of the workshop papers. The workshop
planners describe common metrics in terms of researchers who are pursuing
a line of inquiry that relies on common measures for the variables under
study. Some people mean the development of standard measures that are
driven by policy needs and institutional requirements (e.g., poverty, race,
high school completion). Hauser referred to these as public metrics, but
said that through their use in policy they may take on a life of their own.
Another meaning suggests the development of methods for aligning differ-
ent measures with each other, as illustrated by international benchmarking
of educational measures and approaches to normalizing and transforming
metrics to achieve better comparability. Yet another meaning that is less
explicit is associated with the idea that investigators situate their measures
with respect to others in play.
Although the focus of the workshop is on social science theory,
Bachrach observed, the papers are more concerned with the needs of policy.
She cautioned that how one goes about developing common metrics for
advancing policy may differ from the approach recommended for advanc-
ing theory. Even the definitions captured in the workshop description cover
a very broad set of scenarios depending on how a line of investigation is
interpreted. In her view, perhaps the best contribution that this workshop

Copyright © National Academy of Sciences. All rights reserved.