Page 201 - Statistics II for Dummies

Chapter 10: Sorting Out the Means with Multiple Comparisons 185

This section provides an overview of other multiple comparison procedures
that exist and tells a little bit about each one, including the people who
developed them. Given the dates of when these procedures were developed,
I think you’ll agree with me that the 1950s was the golden age of the multiple
comparison procedure.

Controlling for baloney with
the Bonferroni adjustment

The Bonferroni adjustment (or Bonferroni correction) is a technique used in
a host of situations, not just for multiple comparisons. It was basically
created to stop people from over-analyzing data. There’s a limit to what you
should do when analyzing data; there’s a line that, when crossed, results in
something statisticians call data snooping. And the Bonferroni adjustment
curbs that.

Data snooping is when someone analyzes her data over and over again until
she gets a result that she can say is statistically significant (meaning the result
is said to have been unlikely to have happened by chance; see Chapter 3).
Because the number of tests completed by the data snooper is so high, she’s
likely to find something significant just by chance. And that result is highly
likely to be bogus.
For example, suppose a researcher wants to find out what variable is related
to sales of bedroom slippers. He collects data on everything he can think of,
including the size of people’s feet, the frequency with which they go out to
get the paper in their slippers, and their favorite colors. Not finding anything
significant, he goes on to examine marital status, age, and income.
Still coming up short, he goes out on a limb and looks at hair color, whether
or not the subjects have seen a circus, and where they like to sit on an air-
plane (aisle or window, sir?). Then wouldn’t you know, he strikes gold. Turns
out that, according to his data, people who sit on the aisles on planes are
more likely to buy bedroom slippers than those who sit by the window or in
the middle of a row.

What’s wrong with this picture? Too many tests. Each time the researcher
examines one variable and conducts a test on it, he chooses an α level at
which to conduct the test. (Recall that the α level is the amount of chance
you’re willing to take of rejecting the null hypothesis and making a false
alarm.) As the number of tests increases, the α’s pile up.

16_466469-ch10.indd 185 7/24/09 9:41:43 AM