Page 85 - The Importance of Common Metrics for Advacing Social Science Theory and Research

The Importance of Common Metrics for Advancing Social Science Theory and Research: A Workshop Summary
http://www.nap.edu/catalog/13034.html

FINAL COMMENTS 73

classical and modern test theory. The notion that standardization means
adoption of the same questions is passé. It may be that the latent con-
structs that hearken back to theory are what need to be standardized. He
is most familiar with more parochial politics about the appropriate survey
questions. In contrast, the Patient-Reported Outcomes Measurement In-
formation System lets everyone see their questions in the item bank, and
it transcends the single questionnaire. He questioned whether there could
be a common construct underlying different definitions of poverty if differ-
ent measures of poverty could be subjected to item response theory–type
analysis.
Bohrnstedt reaffirmed the idea that having common concepts does not
mean that the indicators will not change over time. He cited the view, ex-
pressed by Geoff Mulgan, that indicators should change over time because
they are culture- and history-bound, although the concepts should remain
the same.
David Johnson suggested that researchers align themselves with policy
makers and statistical agencies to develop standardized measures, accept-
ing that a perfect measure (e.g., for poverty) is sometimes not possible. He
pointed to work currently being undertaken by the Census Bureau to mea-
sure same-sex marriage (in which decisions are being made today for imple-
mentation in eight years) and vocational education. The Census Bureau has
solicited advice about developing a measure that may not be perfect. As an
indication of progress, he reported that there is a provision of $7.5 million
in the president’s budget that directs the Census Bureau to develop a new
supplemental poverty measure. Robert Pollak was not nearly so optimistic
about the adoption of new poverty measures; changing the definition will
change eligibility for benefits, he said. There are strong constituencies that
will resist this type of change.
Bradburn mentioned three ways in which major indicators become
accepted. First, he noted that the poverty measure is implemented by the
Office of Management and Budget (OMB), whereas the employment rate
1
is implemented by the Bureau of Labor Statistics. If responsibility for a
measure is lodged in the domain of the president’s office (e.g., OMB), it is
likely to be politicized. If responsibility is lodged in one of the statistical
agencies, where the decision makers are generally science professionals, it
will be easier to change the measure (if it is done by the government). Sec-
ond, some very farsighted scientists can set about constructing a measure
before it is needed—an example is the National Assessment of Educational
Progress (NAEP)—and the measure can become adopted as the accepted
measure before it becomes politicized. For NAEP this was largely done by

1 Bradburn attributed this insight to a talk by Rebecca Blank at the American Statistical
Association in August 2009.

Copyright © National Academy of Sciences. All rights reserved.