Page 66 - The Importance of Common Metrics for Advacing Social Science Theory and Research

The Importance of Common Metrics for Advancing Social Science Theory and Research: A Workshop Summary
http://www.nap.edu/catalog/13034.html

54 THE IMPORTANCE OF COMMON METRICS

set of criteria in which the members of the same category do not share any
specific set of features but rather have what Wittgenstein referred to as
“family resemblance.” Such concepts are conglomerations with less precise
boundaries, such as happiness, prestige, social exclusion, and the like.
Definitions depend on their purpose. Bradburn recalled Pollak’s men-
tion of disability and marital status as examples of concepts that could be
defined for a scientific use in order to fit into a theory or be used to make
predictions, or they could be related to policy needs or social descriptive
purposes. He said that concepts can be characterized by explicit definition
(e.g., formulas, such as income = consumption + savings), by implicit defi-
nition (e.g., from scientific uses or attempting axiomatic definitions), or by
operational definition (e.g., IQ). The usual trade-off with respect to com-
mon metrics is between the accuracy of characterization and the purpose
and breadth of applicability.
Once there is a definition, the next concern is that the representation
matches the concept. Thus, concepts referring to specific features like age
or income to some extent can have single-value functions that measure
the values of concern. However, Ballungen concepts are often measured
by indicators or indices. It is often difficult to do much more than simply
count up different indicators, unless some mathematical structure can be
imposed on them. Measurement procedures may combine variables with
different underlying relations to other concepts (e.g., happiness and satis-
faction). Bradburn observed that one of the tensions in the social sciences is
that the more one refines a concept and the more precise one tries to make
it, the more one may lose some of the associations and original meaning,
and comparability across uses may suffer. To consider large numbers of
indicators over time, one ends up reducing or weighting them. Where the
weights come from is of crucial importance to the validity of the measure.
Bradburn saw the need to address these issues of narrowing and redefinition
if a particular set of indicators are to be used for prediction or explanation.
He turned next to two aspects of procedures. One is accuracy in terms
of getting the true value of what one is trying to measure, and the other
is precision or getting a narrow range of estimation. In the social sciences,
researchers do not do much with instrumentation. The issue he identified is
whether survey questions actually measure what one thinks they are mea-
suring. He observed that there is no gold standard for almost all measures
of concepts of interest to social scientists. However, in psychology at least,
this problem was addressed years ago using the multitrait, multimethod
approach—that is, using different measurement modes and different aspects
of the concept to measure something in different ways, which all roughly
converge on the same answer. Such empirical regularities strengthen the
view that the measurement is correct, particularly if it is for scientific pur-

Copyright © National Academy of Sciences. All rights reserved.