Page 84 - The Importance of Common Metrics for Advacing Social Science Theory and Research

The Importance of Common Metrics for Advancing Social Science Theory and Research: A Workshop Summary
http://www.nap.edu/catalog/13034.html

72 THE IMPORTANCE OF COMMON METRICS

George Bohrnstedt repeated some of his challenges to the group to
consider when standardization makes sense. Is there a set of criteria? When
does it not make sense to standardize? What are the costs from not stan-
dardizing? Even when there is benefit to standardization, the incentives to
develop common metrics may be inadequate, especially in some fields in
which academic reputations are built on development of a new method,
concept, or construct.
Norman Bradburn observed that the question of the importance of
standardization has two parts: (1) When does it make a difference and
when is it useful for science? (2) When is it useful for policy issues? On
the science side, when concepts are sufficiently well defined and theory is
sufficiently well formulated, then standardization is important. In terms of
metric or procedures, confidence that the same construct is being measured
is important for advancing theory. He further observed that the lack of
overall theory about psychological processes has led to a reward structure
that places a premium on inventing new measures.
On the policy side, Bradburn elaborated on the use of measures of the
effectiveness of social, economic, or educational policies and the push in
the last decades toward accountability. He commented that any measure
(like the current poverty measure) that is insensitive to the policy lever used
to change it seems to be a bad measure. It would seem that any politician
should want to effectively measure improvements to demonstrate program
success.
Bohrnstedt agreed that science and politics have roles to play. Politics
trumps science. What can the academic community do to mobilize action?
In response to this question, David Grusky commented that society must
choose where it wants politics to intrude in policy decisions. There can
be a cacophony of measures, and politics will intrude in deciding which
measure to feature. Or alternatively, science could advocate for some of-
ficial standard measure, and then politics will intrude on the selection of
that measure. At least the latter is a more transparent process, which gives
scientists an opportunity to provide input.
Bohrnstedt revisited the two measures of intergenerational mobility—
one social, one economic—that society cares immensely about in its ef-
forts to reduce inequality. He believes that having good measures of social
mobility and economic mobility that draw on administrative records is a
good idea. Education is ultimately about a way to reduce inequality and
to facilitate intergenerational mobility. Grusky believes that when there is
more transparency, there is more opportunity for the scientific community
to weigh in at the point of adoption of some sort of official standardized
measurement.
Dennis Fryback questioned what is meant by standardization, spe-
cifically in the health care context, and focused on the difference between

Copyright © National Academy of Sciences. All rights reserved.