Page 59 - The Importance of Common Metrics for Advacing Social Science Theory and Research

The Importance of Common Metrics for Advancing Social Science Theory and Research: A Workshop Summary
http://www.nap.edu/catalog/13034.html

INDICATORS 47

example, determining human capital labor skills in a given area for plant
location or tracking students in school systems—drives how the denomina-
tor and numerator are conceptualized and the choice of methodology.
Turning next to Warren’s presentation, Prewitt reinforced the point
that the policy objective also drives the use of the data set. He suggested
that, in terms of developing common metrics, more conversation is needed
about the differences between administrative data and survey data. Survey
data have the characteristic of being variable rich and case poor due to cost
restrictions. Administrative data have the opposite characteristics: they are
case rich and variable poor. Administrative data are not organized to give
regression analyses about individual-level behavior. By examining the infor-
mation systems of different national governments, the differences between
administrative data and survey data become more apparent. In Europe the
ratio is 85:15 administrative/survey data. In the United States, the ratio is
roughly 80:20 survey/administrative data. If the indicators used are based
in theory, then the theory itself has to connect to a public policy purpose
that is primarily fixed by the administrative agency collecting the data. The
control of the data is in fact with the administrative agency that collects it.
As an aside, Prewitt remarked that digital data will have a significant
impact on the development of standardized measurements. The cost of the
census in the United States is unsustainable, and this will result in a shift
from its current reliance on survey data to increased use of administrative
records and perhaps eventually on digital data. A digital footprint leaves
enormous amounts of data and raises questions about what are proprietary
data.
Prewitt then commented on Mulgan’s presentation describing the evo-
lution of the measurement system, based on the constant interaction be-
tween the quality of the science and the ways in which the data are used. He
said that Mulgan tracked effectively the movement from easily measured
items to more abstract concepts that include subjective well-being, social re-
silience, or social capital. This progression is reflected in policy discussions
about the use of data and the role of the scientific community in influencing
policy makers. It is important, he continued, to control measurement across
the boundaries of a threshold, for example, spending more attention and
money on those “above the threshold” to obtain more funding. Prewitt
acknowledged that social scientists need to live with certain distortions,
but at the same time, he noted, the scientific community has to build in as
many protections as possible so that the system cannot be gamed, as well
as to maintain transparency.
Prewitt emphasized one of Mulgan’s key points about the direction
of social science—the need to incorporate the constituencies affected into
measurement, for example, in the creation of a new disability index. Prewitt
lauded the Oregon benchmark program identified in Mulgan’s presentation,

Copyright © National Academy of Sciences. All rights reserved.