Page 48 - The Importance of Common Metrics for Advacing Social Science Theory and Research

The Importance of Common Metrics for Advancing Social Science Theory and Research: A Workshop Summary
http://www.nap.edu/catalog/13034.html

36 THE IMPORTANCE OF COMMON METRICS

addressing issues like relationships. The latter may not be very
important to policy makers, scientists, or academics, but in fact
is becoming very significant in the day-to-day practice of public
services.
• Indicators are essentially feedback systems to guide decision mak-
ing in public policy, but there is a risk to linking indicators too
closely to policy decisions. Social science needs consistent and
comparable time-series data, whereas the needs of government are
more variable.
• A judgment about indicators needs to address both their construc-
tion and their use. Are they used to constrain fluid actions and
decision making by governments and to assist competitive actions?
Are they assisting effective judgment on conditions of considerable
uncertainty and fuzzy data?
• Both data and the institutions to use them are needed. Having
authoritative public bodies make judgments using standardized
metrics in transparent ways is as important as having the metrics
themselves, and just as important as the recognition that all of
these have, in Mulgan’s words, “limited half lives.”

Mulgan ended by sharing his belief that even the best indicator will be
useful for a time but will then need to replaced and updated, because that
is simply the nature of social knowledge.

STANDARDIZED MEASUREMENT

In his presentation, Robert Pollak raised concerns about the premature
application of standards and the notion that standardization will make for
successful science, rather than the idea that successful science generates
standardization. He gave a number of examples to illustrate his point.
Family structure, in this example marital status, provides an excellent
opportunity to explore the use of standardization, he said, posing several
questions. Does marital status mean that one is legally married? Are cohabi-
tants included? Are couples who are legally married but not living together
included? Does the definition of marital status used as an independent
variable affect the outcome when researchers try to predict educational
outcomes, for example, whether a child will finish high school?
Taking this examination of factors and standardization further, Pollak
raised the question of what it means to complete high school. In other
words, should people with general educational development (or GED)
credentials be treated as high school graduates? Citing work by James
Heckman on labor market effects demonstrating clearly that GED is not
equivalent to high school graduation, Pollak concluded that the question

Copyright © National Academy of Sciences. All rights reserved.