Page 50 - The Importance of Common Metrics for Advacing Social Science Theory and Research

The Importance of Common Metrics for Advancing Social Science Theory and Research: A Workshop Summary
http://www.nap.edu/catalog/13034.html

38 THE IMPORTANCE OF COMMON METRICS

there is no possible response to this kind of question without an underlying
model and theory.
Continuing in the context of disability, Pollak delineated three possible
models or theoretical constructs. One model, which he attributed to Dennis
Fryback, is an appeal to utility that attempts to identify what people actu-
ally value. Another method entails using the theory of disability to predict
the probability of nursing home entry within the next year, on which an
index of disability could be based. His final example was an index that
predicted the medical costs associated with an individual over his or her
lifetime. All of these approaches are different and imply different weights,
items, and methods of calculating disability. Pollak reiterated that, without
an accepted theoretical framework, there is no touchstone for resolving any
of the practical problems that arise in index construction.
Pollak emphasized that his focus is on nontrivial standardization, for
which the measurement choices are really about choosing what is impor-
tant, commenting that this is essentially a scientific question. With non-
trivial standardization, the choices between measurement protocols convey
different information. In his view, measurement without theory often means
measurement using implicit theory. Implicit theory is better when made
explicit, so it can be openly debated. He ended by saying that science is
better done in the open.
Nancy Cartwright expanded on Pollak’s presentation by delineating
three separate avenues by which theory contributes to measurement. The
first she described as “coming up with the representation” using “heavy
theory,” with a lot of assumptions in the theory that are very well worked
out, along with a measure that gives an upper and a lower bound in con-
structing the particular index. A second way is using theory, or at least
empirical regularities that connect the intended quantity with the actual
procedures employed in carrying out measurement. This is done to ensure
that those procedures are measuring the intended concept, especially when
a quantity is measured indirectly via the components of an index, and even
more especially when the components of this index are aggregated into a
single number. The third way is distinguishing among different concepts
going under the same name across a variety of theories. Proper precise
scientific definition and explicit procedures are required when the emphasis
is on making predictions about future behavior or forecasting the effect
of policies. Different studies serving different purposes prescribe different
definitions and procedures, yet they often use the same word. It is important
to keep clear which of these more exact concepts is causally connected with
the outcomes of interest.
Mulgan proposed that, in addition to a different theoretical founda-
tion of a construct, it is important to consider the existence of different
philosophical lenses. For example, he delineated three perspectives related

Copyright © National Academy of Sciences. All rights reserved.