Page 207 - Statistics for Dummies

Chapter 12: Leaving Room for a Margin of Error
Just as important as knowing what the margin of error measures is realizing
what the margin of error does not measure. The margin of error does not
measure anything other than chance variation. That is, it doesn’t measure
any bias or errors that happen during the selection of the participants, the
preparation or conduct of the survey, the data collection and entry process,
or the analysis of the data and the drawing of the final conclusions.
A good slogan to remember when examining statistical results is “garbage in

equals garbage out.” No matter how nice and scientific the margin of error
may look, remember that the formula that was used to calculate it doesn’t
have any idea of the quality of the data that the margin of error is based on. If
the sample proportion or sample mean was based on a biased sample (one
that favored certain people over others), a bad design, bad data-collection
procedures, biased questions, or systematic errors in recording, then calculat-
ing the margin of error is pointless because it won’t mean a thing.
For example, 50,000 people surveyed sounds great, but if they were all visi-
tors to a certain Web site, the margin of error for this result is bogus because 191
the calculation is all based on biased results! In fact, many extremely large
samples are the result of biased sampling procedures. Of course, some
people go ahead and report them anyway, so you have to find out what went
into the formula: good information or garbage? If it turns out to be garbage,
you know what to do about the margin of error. Ignore it. (For more infor-
mation on errors that can take place during a survey or experiment, see
Chapters 16 and 17, respectively.)
The Gallup Organization addresses the issue of what margin of error does
and doesn’t measure in a disclaimer that it uses to report its survey results.
Gallup tells you that besides sampling error, surveys can have additional
errors or bias due to question wording and some of the logistical issues
involved in conducting surveys (such as missing data due to phone numbers
that are no longer current).
This means that even with the best of intentions and the most meticulous
attention to details and process control, stuff happens. Nothing is ever per-
fect. But what you need to know is that the margin of error can’t measure the
extent of those other types of errors. And if a highly credible polling organi-
zation like Gallup admits to possible bias, imagine what’s really going on with
other people’s studies that aren’t nearly as well designed or conducted.

3/25/11 8:15 PM
19_9780470911082-ch12.indd 191 3/25/11 8:15 PM
19_9780470911082-ch12.indd 191