Page 62 - The Importance of Common Metrics for Advacing Social Science Theory and Research

The Importance of Common Metrics for Advancing Social Science Theory and Research: A Workshop Summary
http://www.nap.edu/catalog/13034.html

50 THE IMPORTANCE OF COMMON METRICS

theory aside, Pollak considered the prevailing unemployment and poverty
measures as part of the status quo, which is different from consensus but
may indeed be the consensus.
Karen Jones (Customs and Border Protection) raised the question of
how best to combine good program design with common metrics. She
attended a briefing by the U.S. Government Accountability Office that ad-
dressed practical issues on conducting pure empirical research and how to
mitigate its limitations by using the correct statistics to evaluate the data
gathered. However, she said, there was very little emphasis on common
metrics to evaluate training programs in one field, such as law enforce-
ment. In her field, if something works in a given situation, it is often used
in other situations as long as it meets the minimum criteria for good pro-
gram evaluation design. She questioned how people like her can influence
organizations, like OMB, that continually request adverse impact studies
for training based on arbitrary racial categories.
Referring to the Health and Retirement Study, Willis returned to the
issue of administrative data in connection with surveys. First, an obvious
advantage is a more robust data set resulting from linking representative
survey data with administrative data. Second, this pairing creates an issue
regarding what agency is willing or unwilling to link the data. An agency
that has no policy or policy research aspect will be less inclined to interact
productively with social scientists. Willis argued for a two-way flow of
information, noting that federally funded Research Data Centers that al-
low researchers access to restricted data have benefited from the exchange
between Census Bureau personnel and academics. Prewitt said that ideally
interaction between the producers of administrative data and social scien-
tists would develop in such a way as to yield high-quality data, as well as
better program administration from the resulting data.
Pollak had stated that one should think of a measure in terms of how
the measure works in predicting a certain outcome. Triplett expressed con-
cern about the concept of centering measurement in the political process.
While measurement needs to be of value for analysis, in political and other
contexts, the potential for political or other gaming poses a serious prob-
lem for statistical agencies. The unemployment rate serves as an interesting
example; in the 1970s, it was extremely controversial. The issue was settled
not by theory but in part by the work of Julius Shiskin, who launched sev-
eral different versions of the unemployment rate (called U-1 through U-7)
and showed that they all moved together over the business cycle.
The CPI also generated political debate during Triplett’s tenure at the
Bureau of Labor Statistics, and after. Many of the debates about changing
the CPI focused on technical issues and how to apply the theory underly-
ing the index. Ultimately, this specific debate did not call into question the
integrity of the statistical agency. However, Triplett recalled the creation

Copyright © National Academy of Sciences. All rights reserved.