re: st: Scheffe, Bonferroni and Sidak tests

[David Airey <david.airey@vanderbilt.edu>]

In an ANOVA context, I like the Chapter 5 discussions in the text by 
Maxwell and Delaney, "Designing experiments and analyzing data: A model 
comparison perspective", 2004.

On this list, Roger Newson has provided advice and software to deal 
with multiple comparisons adjustments (findit smileplot). His personal 
site also has good stuff.

Different (posthoc) multiple comparisons procedures are used depending 
on what you are comparing. All pairs? Finding the best treatment? 
Comparing against a control group? You can achieve more power using the 
right test in the right context.

ANOVA in Stata could be improved to at least provide Hsu's, Dunnett's, 
Tukey's, Scheffe's, and Bonferroni'e methods for multiple comparisons 
following any ANOVA. No doubt these procedures can be performed with a 
little programming from the bits and pieces left in memory following 
ANOVA in Stata, but they currently are not easily available to the 
non-programming user.

In psychology, genetics, and neuroscience, multiple test adjustments 
are taken seriously. I'm surprised to hear you saying this from a 
sociology background?

-Dave


> The -oneway- program gives you the option of doing Scheffe, Bonferroni 
> and Sidak tests. Some other commands offer similar options, e.g. 
> -test-. Is there any consensus as to which is best? Are there any 
> situations in which one is clearly preferable to another? My 
> experience has been that results tend to be similar, and the examples 
> from the Stata reference guide also show little difference.
>
> Also, as a practical matter, how often do these tests get done in 
> practice? I see them rarely in the stuff I read and no reviewer has 
> ever asked me to provide this info. I understand the rationale for 
> these tests, but if you take that rationale to its logical conclusion 
> it seems like every coefficient in every model should have its p-value 
> adjusted to reflect the fact that multiple tests are going on. The 
> tests also seem like they can be excessively conservative -- all 
> differences between pairs of groups could be significant at the .05 
> level, but after making one of these adjustments none of the 
> differences could be significant.
*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/